Skip to content

SRE-3150 ci: enable HW Tests on el9.6 #17327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
grom72 wants to merge 40 commits into
base: master
Choose a base branch
from
Draft

SRE-3150 ci: enable HW Tests on el9.6 #17327

grom72 wants to merge 40 commits into from
+292 −68

Conversation

@grom72
Copy link
Contributor

@grom72 grom72 commented Dec 30, 2025

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

ravalsam and others added 30 commits December 30, 2025 15:07
@ravalsam @grom72
SRE-3150 Test: This is Debug PR for NVMe issue
d5cb8e0 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: ravalsam <samirkumar.raval@hpe.com>
@ravalsam @grom72
Reverting it back to original OS.
6c78311 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: ravalsam <samirkumar.raval@hpe.com>
@ravalsam @grom72
Comment node prep NVMe check.
fdb99d2 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: ravalsam <samirkumar.raval@hpe.com>
@ravalsam @grom72
Update test_main_prep_node.sh
9f60be9 
        Skip-unit-tests: true
        Skip-fault-injection-test: true
        Test-tag: test_recycle_pools_hw

    Signed-off-by: ravalsam <samirkumar.raval@hpe.com>

Signed-off-by: Samir Raval <samirkumar.raval@hpe.com>
@ravalsam @grom72
Testing changes.
e4a7c38 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: ravalsam <samirkumar.raval@hpe.com>
@grom72
Break tests repetition after first failure
3f3b111 
Skip-build-el9-gcc: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Default build with pre-set configuration
aecf0b3 
Cancel-prev-build: false

Skip-build-el9-gcc: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Sanity check on el9.5
19a6a3c 
Cancel-prev-build: false

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Skip leap 15 build
8aed634 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Revert "Sanity check on el9.5"
481de85 
This reverts commit e3da5b4.
Cancel-prev-build: false

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Let's try 9.5 again
48bc314 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Cancel-prev-build: false

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true

Test-tag: test_recycle_pools_hw
@grom72
Add parameter to controll HW stages image version
74ebd21 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
Cancel-prev-build: false

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true

Test-tag: test_recycle_pools_hw
@grom72
Stop tests as soon as fault is detected
455f4d5 
Cancel-prev-build: false

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
nvme drives setup script
fde1309 
Doc-only: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Increase max repetition number for cluster sanity check
5b7b932 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Update script to enable two devices per numa node
a77ad8b 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Full NVME devices configuration during node provisioning
5d04c18 
Cancel-prev-build: false

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Speed up test by les repetitions
0ad5d6e 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Avoid error in case of unbinding failure
504ad56 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Run Tests without fix to collect more data
f95e526 
Let's try to run fix with original configuration to confirm that problem
still exists.

Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Run tests with wiped namespaces
b915253 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Revert "Run Tests without fix to collect more data"

This reverts commit 2bae9ce.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Fix: clamav dependency mismatch
1b0f797 
clamav must be removed as required dependency is not avaialble:
"nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit)"

Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Fix: for el9.5 strict order of NVMe devices is needed
36cb784 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Jenkins debugging
168cd13 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Revert "Jenkins debugging"
554b4b2 
This reverts commit e2ba246.
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
fix to debug issue
656ce98 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
fix invalid unmount
91ef413 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Fix - -f does not detect symlink to directories
607065e 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Ignore leap15 builds
e07c277 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Fix - try to rollback few changes to reproduce the problem
0688580 
Cancel-prev-build: false

Quick-build: true
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-test: true
Skip-unit-tests: true
Skip-func-vm: true
Skip-fault-injection-test: true
Skip-test-rpms: true

Test-tag: test_recycle_pools_hw
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/16/execution/node/640/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

@grom72
Fix: script optimization and bug fixing
cd8846f 
Cancel-prev-build: false

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-vm: true
Skip-func-test-vm-valgrind: true
Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-fault-injection-test: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-vmd: false

Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

1 similar comment
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/17/execution/node/614/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

1 similar comment
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/18/execution/node/722/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/18/execution/node/707/log

@grom72
Fix: use custom ctrl id
3325b1e 
Cancel-prev-build: false
Priority: 2
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-vm: true
Skip-func-test-vm-valgrind: true
Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-fault-injection-test: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-vmd: false

Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/696/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/856/log

@grom72
Kick build on el8.8
91cff6f 
Cancel-prev-build: false
Priority: 2
Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-vm: true
Skip-func-test-vm-valgrind: true
Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-fault-injection-test: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-vmd: false

Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 2, 2026

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/20/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 3, 2026

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/670/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 3, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/681/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 3, 2026

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/660/log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@grom72 @daosbuild3 @ravalsam