Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
aacf78e
feat: Rename discovery service
siegfriedweber Dec 10, 2025
726d60f
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Dec 10, 2025
4c69fd1
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Dec 11, 2025
a8c3699
Set initial_cluster_manager_nodes only on cluster_manager nodes; Add …
siegfriedweber Dec 16, 2025
1f15a3a
Fields discoveryServiceExposed and discoveryServiceListenerClass adde…
siegfriedweber Dec 16, 2025
66c9851
Rename opensearch-discovery service to opensearch-seed-nodes and remo…
siegfriedweber Dec 17, 2025
e94df75
wip
siegfriedweber Dec 19, 2025
53e5915
Set seed nodes service scope on the internal TLS volume
siegfriedweber Jan 8, 2026
a99b79e
test: Use the discovery ConfigMap in all tests
siegfriedweber Jan 8, 2026
35c99c5
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 8, 2026
0eb73cc
test(backup-restore): Add the option to disable TLS in S3
siegfriedweber Jan 9, 2026
efbaffc
Publish fully qualified domain names so that the SANs in the TLS cert…
siegfriedweber Jan 13, 2026
111b8d9
test(opensearch-dashboards): Use the OpenSearch discovery ConfigMap f…
siegfriedweber Jan 13, 2026
67a975c
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 13, 2026
701fb52
Fix unit tests
siegfriedweber Jan 13, 2026
58d95b2
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 14, 2026
8969069
test: Use the images built by CI
siegfriedweber Jan 15, 2026
1ffffd4
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 15, 2026
74197eb
chore: Use constant for HTTP port name
siegfriedweber Jan 15, 2026
d492e1e
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 19, 2026
bb5e4af
Fix regexes for attributed string types; Add unit tests for validation
siegfriedweber Jan 20, 2026
5e3e1a0
test: Add unit tests for Port
siegfriedweber Jan 20, 2026
6e05dfe
Remove TODO after successful test
siegfriedweber Jan 20, 2026
e51634c
test(smoke): Remove assertions about PVC statuses
siegfriedweber Jan 20, 2026
feecf60
Add missing start and end tags to regular expressions
siegfriedweber Jan 20, 2026
9102954
Update changelog
siegfriedweber Jan 20, 2026
788176d
docs: Document the discovery ConfigMap
siegfriedweber Jan 21, 2026
6eca6cd
docs: Reference Discovery ConfigMap in the ListenerClass usage guide
siegfriedweber Jan 21, 2026
1c6e096
Remove allow_k8s_contexts from Tiltfile
siegfriedweber Jan 21, 2026
af8558e
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 21, 2026
0966669
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 21, 2026
3b73d2c
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 22, 2026
0291644
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 22, 2026
d6e9dfd
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 22, 2026
66fa16c
test: Make unit tests less verbose
siegfriedweber Jan 22, 2026
2d84854
Merge branch 'main' into feat/service-discovery-and-exposition
siegfriedweber Jan 22, 2026
f39385a
test: Remove OpenSearch 3.4.0 for now
siegfriedweber Jan 22, 2026
b384980
feat: Add OPENSEARCH_HOSTS to discovery ConfigMap
siegfriedweber Jan 22, 2026
04d19ad
docs: Use the discovery ConfigMap in "First Steps"
siegfriedweber Jan 22, 2026
a823c57
docs: Use the discovery ConfigMap in "OpenSearch Dashboards"
siegfriedweber Jan 22, 2026
ad17d19
docs: Fix clippy warnings
siegfriedweber Jan 22, 2026
0c52663
docs: Update image with deployed Kubernetes resources
siegfriedweber Jan 22, 2026
d7c13b7
docs: Update the description of the Kubernetes resources
siegfriedweber Jan 23, 2026
36a6e99
Run pre-commit
siegfriedweber Jan 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,17 @@ All notable changes to this project will be documented in this file.
- Enable the [restart-controller](https://docs.stackable.tech/home/nightly/commons-operator/restarter/), so that the Pods are automatically restarted on config changes ([#97]).
- Configure OpenSearch to publish the fully-qualified domain names of the nodes instead of the IP
addresses, so that TLS certificates can be verified ([#100]).
- Add service discovery and exposition ([#94]):
- Service to set up the cluster renamed to `<cluster-name>-seed-nodes`.
- Discovery service named `<cluster-name>`, added.
The discovery service is used to populate the discovery ConfigMap.
- Discovery ConfigMap named `<cluster-name>`, added.
The ConfigMap contains the keys `OPENSEARCH_HOSTNAME`, `OPENSEARCH_PORT`, `OPENSEARCH_PROTOCOL`
and `OPENSEARCH_HOSTS`. Users should use this information to connect to the cluster.
- Configuration parameter `spec.nodes.roleConfig.discoveryServiceListenerClass` added to set the
ListenerClass for the discovery service.
- Configuration parameter `spec.nodes.roleGroups.<role-group-name>.config.discoveryServiceExposed`
added to expose a role-group via the discovery service.

### Changed

Expand All @@ -24,6 +35,7 @@ All notable changes to this project will be documented in this file.
[#76]: https://github.com/stackabletech/opensearch-operator/pull/76
[#91]: https://github.com/stackabletech/opensearch-operator/pull/91
[#93]: https://github.com/stackabletech/opensearch-operator/pull/93
[#94]: https://github.com/stackabletech/opensearch-operator/pull/94
[#97]: https://github.com/stackabletech/opensearch-operator/pull/97
[#100]: https://github.com/stackabletech/opensearch-operator/pull/100

Expand Down
16 changes: 16 additions & 0 deletions deploy/helm/opensearch-operator/crds/crds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,10 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
discoveryServiceExposed:
description: Determines whether this role group is exposed in the discovery service.
nullable: true
type: boolean
gracefulShutdownTimeout:
description: |-
Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the
Expand Down Expand Up @@ -517,11 +521,19 @@ spec:
x-kubernetes-preserve-unknown-fields: true
roleConfig:
default:
discoveryServiceListenerClass: cluster-internal
podDisruptionBudget:
enabled: true
maxUnavailable: null
description: This is a product-agnostic RoleConfig, which is sufficient for most of the products.
properties:
discoveryServiceListenerClass:
default: cluster-internal
description: The [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) that is used for the discovery service.
maxLength: 253
minLength: 1
pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
type: string
podDisruptionBudget:
default:
enabled: true
Expand Down Expand Up @@ -600,6 +612,10 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
discoveryServiceExposed:
description: Determines whether this role group is exposed in the discovery service.
nullable: true
type: boolean
gracefulShutdownTimeout:
description: |-
Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,25 +75,21 @@ kubectl rollout status --watch statefulset/simple-opensearch-nodes-default --tim
# wait a bit for the port to open
sleep 10

echo "Starting port-forwarding of port 9200"
# tag::opensearch-port-forwarding[]
kubectl port-forward services/simple-opensearch 9200 > /dev/null 2>&1 &
# end::opensearch-port-forwarding[]
PORT_FORWARD_PID=$!
# shellcheck disable=2064 # we want the PID evaluated now, not at the time the trap is
trap "kill $PORT_FORWARD_PID" EXIT
sleep 5

echo "Using the REST API"
# tag::rest-api[]
export CREDENTIALS=admin:AJVFsGJBbpT6mChn

OPENSEARCH_HOST=$(
kubectl get configmap simple-opensearch \
--output=jsonpath='{.data.OPENSEARCH_HOSTS}'
)

curl \
--insecure \
--user $CREDENTIALS \
--request PUT \
--json '{"name": "Stackable"}' \
https://localhost:9200/sample_index/_doc/1
"$OPENSEARCH_HOST/sample_index/_doc/1"

# Output:
# {"_index":"sample_index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
Expand All @@ -102,7 +98,7 @@ curl \
--insecure \
--user $CREDENTIALS \
--request GET \
https://localhost:9200/sample_index/_doc/1
"$OPENSEARCH_HOST/sample_index/_doc/1"

# Output:
# {"_index":"sample_index","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"name": "Stackable"}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,25 +75,21 @@ kubectl rollout status --watch statefulset/simple-opensearch-nodes-default --tim
# wait a bit for the port to open
sleep 10

echo "Starting port-forwarding of port 9200"
# tag::opensearch-port-forwarding[]
kubectl port-forward services/simple-opensearch 9200 > /dev/null 2>&1 &
# end::opensearch-port-forwarding[]
PORT_FORWARD_PID=$!
# shellcheck disable=2064 # we want the PID evaluated now, not at the time the trap is
trap "kill $PORT_FORWARD_PID" EXIT
sleep 5

echo "Using the REST API"
# tag::rest-api[]
export CREDENTIALS=admin:AJVFsGJBbpT6mChn

OPENSEARCH_HOST=$(
kubectl get configmap simple-opensearch \
--output=jsonpath='{.data.OPENSEARCH_HOSTS}'
)

curl \
--insecure \
--user $CREDENTIALS \
--request PUT \
--json '{"name": "Stackable"}' \
https://localhost:9200/sample_index/_doc/1
"$OPENSEARCH_HOST/sample_index/_doc/1"

# Output:
# {"_index":"sample_index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}
Expand All @@ -102,7 +98,7 @@ curl \
--insecure \
--user $CREDENTIALS \
--request GET \
https://localhost:9200/sample_index/_doc/1
"$OPENSEARCH_HOST/sample_index/_doc/1"

# Output:
# {"_index":"sample_index","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"name": "Stackable"}}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
---
opensearchHosts: https://simple-opensearch-nodes-default.default.svc.cluster.local:9200
image:
repository: oci.stackable.tech/sdp/opensearch-dashboards
tag: 3.1.0-stackable0.0.0-dev
Expand All @@ -23,6 +22,11 @@ config:
cookie:
secure: true
extraEnvs:
- name: OPENSEARCH_HOSTS
valueFrom:
configMapKeyRef:
name: simple-opensearch
key: OPENSEARCH_HOSTS
- name: OPENSEARCH_PASSWORD
valueFrom:
secretKeyRef:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
---
opensearchHosts: https://simple-opensearch-nodes-default.default.svc.cluster.local:9200
image:
repository: oci.stackable.tech/sdp/opensearch-dashboards
tag: 3.1.0-stackable{{ versions.opensearch }}
Expand All @@ -23,6 +22,11 @@ config:
cookie:
secure: true
extraEnvs:
- name: OPENSEARCH_HOSTS
valueFrom:
configMapKeyRef:
name: simple-opensearch
key: OPENSEARCH_HOSTS
- name: OPENSEARCH_PASSWORD
valueFrom:
secretKeyRef:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ spec:
image:
productVersion: 3.1.0
nodes:
roleConfig:
discoveryServiceListenerClass: external-stable
roleGroups:
default:
replicas: 3
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 0 additions & 11 deletions docs/modules/opensearch/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,6 @@ You can do so with this command:
include::example$getting_started/getting_started.sh[tag=await-cluster]
----

== Connecting to the HTTP endpoint

Once the OpenSearch nodes are created, you can use the REST API of OpenSearch.

To forward the HTTP port (`9200`) to localhost, run:

[source,bash]
----
include::example$getting_started/getting_started.sh[tag=opensearch-port-forwarding]
----

== Using the REST API

You can use the REST API as follows:
Expand Down
11 changes: 6 additions & 5 deletions docs/modules/opensearch/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -41,17 +41,18 @@ It helps you tune your cluster to your needs by configuring xref:usage-guide/sto

=== Kubernetes resources

Based on the custom resources you define, the operator creates ConfigMaps, StatefulSets and Services.
Based on the custom resources you define, the operator creates ConfigMaps, StatefulSets, Services and so on.

image::opensearch_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the operator]

The diagram above depicts all the Kubernetes resources created by the operator, and how they relate to each other.

For every xref:concepts:roles-and-role-groups.adoc#role-groups[role group] you define, the operator creates a StatefulSet with the amount of replicas defined in the role group.
For every role group, a Service is created, as well as one for the whole cluster that references the cluster manager nodes.
What should be highlighted, is the xref:reference/discovery.adoc[discovery ConfigMap] which is named the same as the OpenSearchCluster.
It references the Service that should be used to connect to the cluster.

Additionally, a ConfigMap is created for each role group.
These ConfigMaps contain configuration files like `opensearch.yml`.
For every xref:concepts:roles-and-role-groups.adoc#role-groups[role group] you define, the operator deploys OpenSearch as a StatefulSet with the amount of replicas defined in the role group.
The pods of a StatefulSet use the configuration from the role group ConfigMap, i.e. they all use the same OpenSearch node roles, e.g. `cluster-manager` or `data`, and xref:opensearch:usage-guide/storage-resource-configuration.adoc[resource configurations].
If you want dedicated `cluster-manager` and `data` nodes, you just define two role groups with according configurations as described in xref:opensearch:usage-guide/node-roles.adoc[].

== Supported versions

Expand Down
100 changes: 100 additions & 0 deletions docs/modules/opensearch/pages/reference/discovery.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
:clusterName: simple-opensearch
:namespace: stackable
:exampleDiscoveryServiceListenerClass: external-unstable
:exampleOpensearchProtocol: https
:exampleOpensearchIp: 10.104.213.49
:exampleOpensearchPort: 31315

= Discovery
:page-aliases: discovery.adoc

The Stackable Operator for OpenSearch publishes a xref:concepts:service_discovery.adoc[service discovery ConfigMap] which exposes a client configuration bundle that allows access to the OpenSearch cluster.

The bundle includes the connection parameters to access the OpenSearch cluster.
These parameters may be used by other operators or tools to configure their products with access to OpenSearch.

== Example

Given the following OpenSearch cluster:

[source,yaml,subs="normal,callouts"]
----
apiVersion: opensearch.stackable.tech/v1alpha1
kind: OpenSearchCluster
metadata:
name: {clusterName} # <1>
namespace: {namespace} # <2>
spec:
clusterConfig:
tls:
serverSecretClass: tls # <3>
nodes:
roleConfig:
discoveryServiceListenerClass: {exampleDiscoveryServiceListenerClass} # <4>
roleGroups:
cluster-manager:
config:
discoveryServiceExposed: true # <5>
nodeRoles:
- cluster_manager
data:
config:
discoveryServiceExposed: false # <6>
nodeRoles:
- ingest
- data
- remote_cluster_client
----
<1> The name of the OpenSearch cluster which is also the name of the created discovery ConfigMap.
<2> The namespace of the cluster and the discovery ConfigMap.
<3> Whether a `serverSecretClass` is set or not, determines the value of the `OPENSEARCH_PROTOCOL` key in the discovery ConfigMap.
<4> The xref:listener-operator:listenerclass.adoc[ListenerClass] that is used for the discovery service.
<5> The `cluster-manager` role group is exposed in the discovery service.
<6> The `data` role group is not exposed in the discovery service.

The resulting discovery ConfigMap is `{namespace}/{clusterName}`.

== Contents

The `{namespace}/{clusterName}` discovery ConfigMap contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:

`OPENSEARCH_HOSTNAME`::
====
Contains the hostname or IP of the service that references the exposed role groups.

In case, `discoveryServiceListenerClass` was set to `cluster-internal`, the following hostname will be set:

[subs="normal"]
{clusterName}.{namespace}.svc.cluster.local

If `discoveryServiceListenerClass` was set to `{exampleDiscoveryServiceListenerClass}`, the content could look like:

[subs="normal"]
{exampleOpensearchIp}
====

`OPENSEARCH_PORT`::
====
Contains the port of the service that references the exposed role groups.

Depending on the `discoveryServiceListenerClass`, the port will be either the default HTTP port 9200 or a NodePort:

[subs="normal"]
{exampleOpensearchPort}
====

`OPENSEARCH_PROTOCOL`::
====
Contains either `http` or `https`, depending on whether a `serverSecretClass` is configured:

[subs="normal"]
{exampleOpensearchProtocol}
====

`OPENSEARCH_HOSTS`::
====
Contains the URL of the service that references the exposed role groups.

[subs="normal"]
{exampleOpensearchProtocol}://{exampleOpensearchIp}:{exampleOpensearchPort}
====
1 change: 1 addition & 0 deletions docs/modules/opensearch/pages/reference/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@
Consult the reference documentation section to find exhaustive information on:

* Descriptions and default values of all properties in the CRDs used by this operator in the xref:reference/crds.adoc[].
* The properties in the xref:reference/discovery.adoc[Discovery ConfigMap].
* The xref:reference/commandline-parameters.adoc[] and xref:reference/environment-variables.adoc[] accepted by the operator.
11 changes: 8 additions & 3 deletions docs/modules/opensearch/pages/usage-guide/listenerclass.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
= Service exposition with ListenerClasses
:description: Configure OpenSearch service exposure with ListenerClasses: cluster-internal, external-unstable, or external-stable.

The operator deploys a xref:listener-operator:listener.adoc[Listener] for OpenSearch role-groups.
[NOTE]
====
While the listeners described here provide access to individual role groups, the xref:opensearch:reference/discovery.adoc[Discovery ConfigMap] is the recommended approach for general access to the OpenSearch cluster.
====

The operator deploys a xref:listener-operator:listener.adoc[Listener] for each OpenSearch role group.
The listener defaults to only being accessible from within the Kubernetes cluster, but this can be changed by setting `.spec.nodes.roleGroups.\{role-group-name}.config.listenerClass`:

[source,yaml]
Expand All @@ -13,5 +18,5 @@ spec:
config:
listenerClass: external-stable # <1>
----
<1> Specify a ListenerClass, such as `external-stable`, `external-unstable`, or `cluster-internal` (the default setting is `cluster-internal`) at role-group level.
This can be set for all role-groups individually.
<1> Specify a ListenerClass, such as `external-stable`, `external-unstable`, or `cluster-internal` (the default setting is `cluster-internal`) at role group level.
This can be set for all role groups individually.
Loading