Skip to content

Commit 887ea23

Browse files
aliceinwirealiceinwire
authored
Merge pull request #242 from aliceinwire/update_docs_validate
(docs) Add maestro validate
2 parents f68bf99 + 96451e6 commit 887ea23

File tree

1 file changed

+271
-0
lines changed

1 file changed

+271
-0
lines changed

‎docs/validate.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# `kci-dev maestro validate` — Maestro/Dashboard Consistency Checker
2+
3+
Validate KernelCI dashboard results against Maestro pipeline outputs.
4+
5+
This command group compares **builds** and **boots** recorded in the KernelCI dashboard with the corresponding Maestro results (KCIDB origin `maestro` by default). It flags count mismatches, lists missing IDs, and detects status mismatches.
6+
7+
---
8+
9+
## TL;DR
10+
11+
```bash
12+
# Validate latest builds for all trees seen in the last N days
13+
kci-dev maestro validate builds --all-checkouts --days 7
14+
15+
# Validate latest boots for all trees seen in the last N days
16+
kci-dev maestro validate boots --all-checkouts --days 7
17+
18+
# Validate a specific checkout (builds)
19+
kci-dev maestro validate builds --giturl <url> --branch <branch> --commit <commit>
20+
21+
# Validate a specific checkout (boots)
22+
kci-dev maestro validate boots --giturl <url> --branch <branch> --commit <commit>
23+
24+
# Compare build history across multiple checkouts for the same tree/branch
25+
kci-dev maestro validate builds --history --giturl <url> --branch <branch> --days 14
26+
27+
# Show results as a table (otherwise prints a simple list)
28+
kci-dev maestro validate builds --table-output ...
29+
kci-dev maestro validate boots --table-output ...
30+
```
31+
32+
---
33+
34+
## Usage
35+
36+
```text
37+
kci-dev maestro validate [SUBCOMMAND]
38+
39+
Subcommands:
40+
builds Validate build results
41+
boots Validate boot results
42+
```
43+
44+
Invoking `kci-dev maestro validate` without a subcommand prints help and exits with code `0`.
45+
46+
---
47+
48+
## Shared Concepts
49+
50+
* **Origin**: KCIDB origin to pull dashboard “trees” from. Defaults to `maestro`.
51+
* **All checkouts vs specific checkout**:
52+
53+
* `--all-checkouts` validates the **latest** results for all trees observed in the last `--days` days.
54+
* A **specific checkout** is identified by `--giturl`, `--branch`, and `--commit`.
55+
If not provided, the tool can infer from `--git-folder` (or use `--latest`).
56+
* **Arch filter**: `--arch <arch>` narrows both Maestro and dashboard queries.
57+
* **Output modes**:
58+
59+
* **Simple list (default)**: concise, includes per-commit links to missing items.
60+
* **Table**: `--table-output` renders a grid with counts, flags, and IDs.
61+
62+
---
63+
64+
## Status Mapping & Mismatch Rules
65+
66+
### Builds
67+
68+
* Maestro `result` → Dashboard `status`
69+
70+
* `pass` → `PASS`
71+
* `fail` → `FAIL`
72+
* `incomplete` → `ERROR`
73+
* Retries excluded unless `retry_counter == 3` (to avoid counting transient retries).
74+
* A **status mismatch** is reported if the mapped status doesn’t match the dashboard.
75+
76+
### Boots
77+
78+
* Maestro `result`:
79+
80+
* `pass` → `PASS`
81+
* `fail` → `FAIL`
82+
* `incomplete` → mapped using `data.error_code`:
83+
84+
* **MISS** if error_code ∈ {`Bug`, `Configuration`, `invalid_job_params`, `Job`, `job_generation_error`, `ObjectNotPersisted`, `RequestBodyTooLarge`, `submit_error`, `Unexisting permission codename.`, `kbuild_internal_error`}
85+
* **ERROR** if error_code ∈ {`Canceled`, `Infrastructure`, `LAVATimeout`, `MultinodeTimeout`, `Test`}
86+
* Boot retries excluded unless `retry_counter == 3`.
87+
* A **status mismatch** is reported when the mapped result disagrees with the dashboard.
88+
89+
---
90+
91+
## Subcommand: `builds`
92+
93+
Validate dashboard build results against Maestro.
94+
95+
```bash
96+
kci-dev maestro validate builds [OPTIONS]
97+
```
98+
99+
### Options
100+
101+
| Option | Type | Default | Description |
102+
| ----------------- | ---- | --------: | ------------------------------------------------------------------------------------------------------------------------- |
103+
| `--all-checkouts` | flag | | Validate latest builds for all available trees within `--days`. **Mutually exclusive** with `--giturl/--branch/--commit`. |
104+
| `--days` | int | `7` | Time window (days) for selecting trees/checkouts. |
105+
| `--origin` | str | `maestro` | KCIDB origin for tree discovery. |
106+
| `--giturl` | str | | Kernel tree Git URL. |
107+
| `--branch` | str | | Branch name. |
108+
| `--commit` | str | | Commit SHA or tag. |
109+
| `--git-folder` | path | | Local repo path used to infer missing Git params. |
110+
| `--latest` | flag | | Use the latest available results (when not specifying `--commit`). |
111+
| `--arch` | str | | Architecture filter. |
112+
| `--history` | flag | | Validate **build count consistency across multiple checkouts** for the same tree/branch within `--days`. |
113+
| `--verbose` | flag | `false` | Print detailed lists of missing items (JSON). |
114+
| `--table-output` | flag | `false` | Show a table (otherwise prints a simple list). |
115+
116+
> ❗ `--all-checkouts` **cannot** be combined with `--giturl`, `--branch`, or `--commit`.
117+
> If `--history` is used with `--all-checkouts`, history is computed for **each tree** seen within `--days`.
118+
> If `--history` is used without `--all-checkouts`, you must specify the tree via `--giturl` and `--branch`.
119+
120+
### Output (Builds)
121+
122+
* **Counts**: Maestro vs Dashboard, with a comparison flag:
123+
124+
* `✅` counts match
125+
* `❌` counts differ
126+
* **Missing IDs**: Item IDs present on one side but not the other.
127+
Missing dashboard items are printed as viewer links:
128+
129+
```
130+
https://api.kernelci.org/viewer?node_id=<id>
131+
```
132+
* **Status mismatches**: List of build IDs where the mapped Maestro status != Dashboard status.
133+
134+
#### Example: Table Output
135+
136+
```text
137+
+------------------+------------------------------------------+---------+------------+-------------------------+---------------------+-------------------------------+
138+
| tree/branch | Commit | Maestro | Dashboard | Build count | Missing build IDs | Builds with |
139+
| | | builds | builds | comparison | | status mismatch |
140+
+==================+==========================================+=========+============+=========================+=====================+===============================+
141+
| linux/mainline | 1a2b3c4d5e6f... | 120 | 120 | ✅ | [] | [] |
142+
| linux/stable-6.6 | f00baa... | 118 | 116 | ❌ | [<ids>...] | [<ids>...] |
143+
+------------------+------------------------------------------+---------+------------+-------------------------+---------------------+-------------------------------+
144+
```
145+
146+
#### Example: Simple List (default)
147+
148+
```text
149+
• linux/mainline: ✅
150+
Commit: 1a2b3c4d5e6f...
151+
152+
• linux/stable-6.6: ❌
153+
Commit: f00baa...
154+
Missing builds: 2
155+
- https://api.kernelci.org/viewer?node_id=<id1>
156+
- https://api.kernelci.org/viewer?node_id=<id2>
157+
```
158+
159+
---
160+
161+
## Subcommand: `boots`
162+
163+
Validate dashboard boot results against Maestro.
164+
165+
```bash
166+
kci-dev maestro validate boots [OPTIONS]
167+
```
168+
169+
### Options
170+
171+
| Option | Type | Default | Description |
172+
| ----------------- | ---- | --------: | ------------------------------------------------------------------------------------------------------------------------ |
173+
| `--all-checkouts` | flag | | Validate latest boots for all available trees within `--days`. **Mutually exclusive** with `--giturl/--branch/--commit`. |
174+
| `--days` | int | `7` | Time window (days) for selecting trees/checkouts. |
175+
| `--origin` | str | `maestro` | KCIDB origin for tree discovery. |
176+
| `--giturl` | str | | Kernel tree Git URL. |
177+
| `--branch` | str | | Branch name. |
178+
| `--commit` | str | | Commit SHA or tag. |
179+
| `--git-folder` | path | | Local repo path used to infer missing Git params. |
180+
| `--latest` | flag | | Use the latest available results (when not specifying `--commit`). |
181+
| `--arch` | str | | Architecture filter. |
182+
| `--verbose` | flag | `false` | Print detailed lists of missing items (JSON). |
183+
| `--table-output` | flag | `false` | Show a table (otherwise prints a simple list). |
184+
185+
### Output (Boots)
186+
187+
* **Counts**: Maestro vs Dashboard boots, with `✅/❌` flag.
188+
* **Missing boot IDs** with viewer links (same format as builds).
189+
* **Status mismatches** using the boot-specific mapping (including `MISS`/`ERROR` for `incomplete` with known `error_code` categories).
190+
191+
---
192+
193+
## History Mode (Builds Only)
194+
195+
`--history` evaluates **count consistency across multiple recent checkouts** of the same tree/branch within `--days`.
196+
Output summarizes each commit under the tree/branch and flags commits where counts differ, along with missing IDs.
197+
198+
Example (simple list):
199+
200+
```text
201+
linux/mainline:
202+
Commit 1a2b3c4d5e6f: ✅
203+
Commit 9c8d7e6f5a4b: ❌
204+
Missing builds: 3
205+
- https://api.kernelci.org/viewer?node_id=<idA>
206+
- https://api.kernelci.org/viewer?node_id=<idB>
207+
- https://api.kernelci.org/viewer?node_id=<idC>
208+
```
209+
210+
---
211+
212+
## Inference Helpers
213+
214+
When `--giturl/--branch/--commit` are omitted:
215+
216+
* `--git-folder` can be used to infer values from a local clone.
217+
* `--latest` selects the latest available results for the inferred tree/branch.
218+
219+
---
220+
221+
## Exit & Error Behavior
222+
223+
* Running `kci-dev maestro validate` without a subcommand prints help and exits `0`.
224+
* If dashboard queries fail or are aborted, the tool prints an error and **skips** stats for that item.
225+
* If a dashboard responds with “No builds/boots available,” the tool treats the dashboard count as `0` for that checkout.
226+
227+
---
228+
229+
## Examples (Copy/Paste)
230+
231+
```bash
232+
# All builds for all trees in last 3 days (table view)
233+
kci-dev maestro validate builds --all-checkouts --days 3 --table-output
234+
235+
# Specific checkout (boots) for x86_64 only
236+
kci-dev maestro validate boots \
237+
--giturl https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git \
238+
--branch master \
239+
--commit <sha> \
240+
--arch x86_64
241+
242+
# Build history for a stable branch over the last 14 days
243+
kci-dev maestro validate builds --history \
244+
--giturl https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git \
245+
--branch linux-6.6.y \
246+
--days 14 \
247+
--table-output
248+
249+
# Infer from local repo, pick latest
250+
kci-dev maestro validate builds --git-folder /path/to/linux --latest
251+
```
252+
253+
---
254+
255+
## Notes & Gotchas
256+
257+
* **Mutual exclusion**: `--all-checkouts` cannot be combined with any of `--giturl/--branch/--commit`.
258+
* **Retries**: To avoid double-counting, build/boot retries are excluded unless `retry_counter == 3`.
259+
* **Viewer links**: Missing items are printed with `https://api.kernelci.org/viewer?node_id=<id>` for quick inspection.
260+
* **Verbose JSON**: Use `--verbose` to print the full JSON of missing items for debugging.
261+
* **Formats**: The table view uses `tabulate`’s `simple_grid` format with reasonable column width caps.
262+
263+
---
264+
265+
## See Also
266+
267+
* `kci-dev results` (for raw result listing and filtering)
268+
* `kci-dev maestro validate builds --help`
269+
* `kci-dev maestro validate boots --help`
270+
271+
> This page documents the CLI behavior surfaced by `kcidev.subcommands.maestro.validate.{builds,boots}` and their helpers.

0 commit comments

Comments
 (0)