Dapr state store clickhouse #3675

middt · 2025-02-23T04:10:23Z

Description

This PR adds a new state store component for ClickHouse, a column-oriented database management system. The ClickHouse state store component provides the following features:

Basic CRUD operations for state management
Support for TTL (Time-To-Live)
ETag support for optimistic concurrency
Bulk operations support
Username/password authentication
Uses ReplacingMergeTree engine for efficient updates

Key implementation details:

Uses the official ClickHouse Go driver
Implements the full state.Store interface
Handles connection management and cleanup
Includes comprehensive integration tests
Follows Dapr's component development guidelines

Implementation Details

The component includes:

State store implementation (clickhouse.go)
- Full CRUD operations
- Bulk operations
- TTL support
- ETag support
- Authentication support
Tests (clickhouse_test.go)
- Integration tests for CRUD operations
- Metadata validation tests
- Authentication tests
- Cleanup handling

Configuration options:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
  namespace: default
spec:
  type: state.clickhouse
  version: v1
  metadata:
  - name: clickhouseURL
    value: "tcp://localhost:9000"
  - name: databaseName
    value: "dapr_state"
  - name: tableName
    value: "state_store"
  - name: username
    value: "default"
  - name: password
    value: ""

Checklist

Code compiles correctly
Created/updated tests
Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Testing Done

Unit tests for metadata validation
Integration tests for:
- Basic CRUD operations
- TTL functionality
- ETag support
- Bulk operations
- Authentication
- Error handling

Tests were run against ClickHouse v23.8 using the official Go driver.

Additional Notes

The component uses ReplacingMergeTree engine for better update performance
Includes proper cleanup in tests to avoid test pollution
Follows ClickHouse best practices for table design
Implements proper connection management and cleanup

Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

JoshVanL · 2025-03-13T14:58:17Z

Thanks @middt, implementation generally looks good to me!

Would it be possible to add conformance tests for this too? It seems like we should be able to spin up a click house docker compose.

…tation Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

…tion handling Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

… connection handling Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

middt · 2025-03-14T07:04:22Z

Thank you for the feedback! @JoshVanL I've implemented the conformance tests for the ClickHouse state store in this PR:

Created a Docker Compose setup in /tests/certification/state/clickhouse/docker-compose.yml that spins up a ClickHouse server with proper configuration
Added a conformance test file in /tests/certification/state/clickhouse/clickhouse_test.go that verifies all state store operations
Created configuration files in /tests/config/state/clickhouse/clickhouse.yml for the test environment
Added ClickHouse to the state store tests configuration in /tests/config/state/tests.yml

The tests verify all the key functionality including:

Basic CRUD operations
ETag support for optimistic concurrency
Error handling for empty and non-existent keys
Feature reporting

I've also addressed authentication issues by properly configuring username and password in both the Docker Compose setup and the state store implementation.

All unit tests are now passing, confirming that the implementation works correctly with the ClickHouse server.

Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

JoshVanL · 2025-04-09T19:18:24Z

Thanks @middt, I think the only thing left is to do a go mod tidy and it should be good to go 🙂

Appreciate the work on this!

…tests Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

github-actions · 2025-11-08T16:47:15Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

github-actions · 2025-11-15T16:55:19Z

This pull request has been automatically closed because it has not had activity in the last 37 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

sicoyle

few comments. Nice job 👏

sicoyle · 2025-12-09T20:58:33Z

state/clickhouse/clickhouse.go

@@ -0,0 +1,435 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:03:02Z

state/clickhouse/clickhouse.go

+	// Create database if not exists
+	// Note: Database and table names are validated during metadata parsing
+	// and come from trusted configuration, so direct string concatenation is acceptable here
+	createDBQuery := "CREATE DATABASE IF NOT EXISTS " + c.config.DatabaseName


we need to make sure this behavior is documented. Have you created a docs PR for these changes by chance? We try to have a docs PR open before merging new components please 🙏

sicoyle · 2025-12-09T21:09:18Z

state/clickhouse/clickhouse.go

+	if config.DatabaseName == "" {
+		return config, errors.New("ClickHouse database name is missing")
+	}
+
+	if config.TableName == "" {


are there fixed char limits on these we should check for by chance?

@sicoyle looks like it's here.
@middt can you add a length check as well?

sicoyle · 2025-12-09T21:10:09Z

state/clickhouse/clickhouse_integration_test.go

@@ -0,0 +1,569 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:11:54Z

state/clickhouse/clickhouse_test.go

@@ -0,0 +1,240 @@
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:36:03Z

state/clickhouse/metadata.yaml

+  - title: "Username and password"
+    description: "Authenticate using username and password."
+    metadata:
+      - name: Username
+        type: string
+        required: false
+        description: |
+          Username for ClickHouse host. Defaults to empty.
+        example: "default"
+        default: ""
+      - name: Password
+        type: string
+        required: false
+        sensitive: true
+        description: |
+          Password for ClickHouse host. No default. Use secretKeyRef for
+          secret reference
+        example: "mypassword"
+        default: ""


this says that none of the fields for auth are required. Can you please confirm if this is this true that we can connect without a username and password?

sicoyle · 2025-12-09T21:46:34Z

state/clickhouse/metadata.yaml

+        example: "mypassword"
+        default: ""
+metadata:
+  - name: ClickhouseURL


can you pls update these all to be camelcase?

sicoyle · 2025-12-09T21:46:50Z

tests/certification/state/clickhouse/clickhouse_test.go

+// +build conftests
+
+/*
+Copyright 2021 The Dapr Authors


Suggested change

Copyright 2021 The Dapr Authors

Copyright 2025 The Dapr Authors

sicoyle · 2025-12-09T21:49:52Z

@JoshVanL did you have any other feedback here by chance?

CasperGN · 2025-12-15T08:46:40Z

@sicoyle I think this one is actually super interesting to get in. I've seen an uptick in Clickhouse lately so supporting this would be beneficial imo.
@middt please address the comments so we can get this one merged 😄

… use camelCase for metadata Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

middt · 2025-12-15T11:49:34Z

Hi @sicoyle @CasperGN! Thank you for the thorough review! I've addressed all the feedback in the latest commit (1be750b):

✅ Changes Made:

Copyright Year Updated (2021 → 2025)
- Updated in all ClickHouse-related files (clickhouse.go, clickhouse_test.go, clickhouse_integration_test.go, and tests/certification/state/clickhouse/clickhouse_test.go)
Added Length Checks for Database and Table Names
- Added a maxIdentifierLength constant set to 256 characters (based on ClickHouse documentation)
- Added validation in parseAndValidateMetadata() to return an error if names exceed the limit
Updated Metadata Field Names to camelCase
- ClickhouseURL → clickhouseUrl
- DatabaseName → databaseName
- TableName → tableName
- Username → username
- Password → password
- Added proper mapstructure tags in the Go struct for proper binding
- Updated all test files and config files to use the new camelCase names
Documentation PR
- Yes, a docs PR was already created: Add Clickhouse state store docs v1.16 docs#4697 (targeting v1.16 branch)
Username/Password Authentication
- Confirmed: ClickHouse can be configured to run without authentication (using the default user with no password). This is common in development environments. In production, authentication should be configured. The fields are correctly marked as required: false in the metadata schema.

All tests are passing and linting is clean. Ready for re-review! 🚀

CasperGN · 2025-12-15T12:01:30Z

state/clickhouse/clickhouse.go

 }

+// maxIdentifierLength is the maximum length for database and table names in ClickHouse
+const maxIdentifierLength = 256


Per their docs and this issue they reference a length of 206 and 250 respectively - are you positive about the 256 limit?

…test The certification test had '//go:build conftests' but the CI runs with '-tags=certtests,unit'. Other certification tests don't use build constraints at all. Removed the build constraint to match the pattern used by other certification tests in the repository. Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

middt added 18 commits November 13, 2024 14:35

Added clickhouse state store

798489e

Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

Merge branch 'main' into dapr-state-store-clickhouse

d4ba4a7

Merge branch 'main' into dapr-state-store-clickhouse

a14a0b4

Merge branch 'main' into dapr-state-store-clickhouse

16af996

Merge branch 'main' into dapr-state-store-clickhouse

47b4b8b

Merge branch 'main' into dapr-state-store-clickhouse

2292ac1

Merge branch 'main' into dapr-state-store-clickhouse

f141667

Merge branch 'main' into dapr-state-store-clickhouse

cc80a66

Merge branch 'main' into dapr-state-store-clickhouse

40590ba

Merge branch 'main' into dapr-state-store-clickhouse

8d795ba

Merge branch 'main' into dapr-state-store-clickhouse

1137d31

Merge branch 'main' into dapr-state-store-clickhouse

8cfc8c1

Merge branch 'main' into dapr-state-store-clickhouse

97ab7c7

Merge branch 'main' into dapr-state-store-clickhouse

3f8679f

Merge branch 'main' into dapr-state-store-clickhouse

d541fda

Merge branch 'main' into dapr-state-store-clickhouse

3e2b6b0

Merge branch 'main' into dapr-state-store-clickhouse

8c81e28

Merge branch 'main' into dapr-state-store-clickhouse

43e4645

middt requested review from a team as code owners February 23, 2025 04:10

Merge branch 'main' into dapr-state-store-clickhouse

7c3242e

middt added 3 commits March 14, 2025 10:02

Add conformance tests for ClickHouse state store and improve implemen…

7a33ea7

…tation Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

Improve ClickHouse state store with authentication support and connec…

0fcdce3

…tion handling Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

Improve ClickHouse state store with authentication support and proper…

2c8934c

… connection handling Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

middt force-pushed the dapr-state-store-clickhouse branch from 1b2a8de to 2c8934c Compare March 14, 2025 07:02

middt added 2 commits March 14, 2025 10:05

Merge branch 'main' into dapr-state-store-clickhouse

441acfd

Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

Merge branch 'main' into dapr-state-store-clickhouse

c7c2507

Signed-off-by: Mehmet TOSUN <93265833+middt@users.noreply.github.com>

Fix ClickHouse certification tests and add comprehensive integration …

e35c604

…tests Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

middt force-pushed the dapr-state-store-clickhouse branch from 4f3e99d to e35c604 Compare September 1, 2025 07:33

middt and others added 8 commits September 2, 2025 15:56

Fix ClickHouse conformance test URL format

004d30a

Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

Merge branch 'main' into dapr-state-store-clickhouse

e221875

Merge branch 'main' into dapr-state-store-clickhouse

1e07076

Merge branch 'main' into dapr-state-store-clickhouse

ffc1a92

Merge branch 'main' into dapr-state-store-clickhouse

0628cd6

Merge branch 'main' into dapr-state-store-clickhouse

1de3b44

Merge branch 'main' into dapr-state-store-clickhouse

10423ae

Merge branch 'main' into dapr-state-store-clickhouse

ab6b744

github-actions bot added the stale label Nov 8, 2025

github-actions bot closed this Nov 15, 2025

yaron2 reopened this Nov 15, 2025

github-actions bot removed the stale label Nov 15, 2025

dapr-bot added 4 commits November 15, 2025 10:08

Merge branch 'main' into dapr-state-store-clickhouse

233a6ec

Merge branch 'main' into dapr-state-store-clickhouse

19ec6de

Merge branch 'main' into dapr-state-store-clickhouse

d7ccb49

Merge branch 'main' into dapr-state-store-clickhouse

322e946

sicoyle reviewed Dec 9, 2025

View reviewed changes

Address PR review comments: update copyright year, add length checks,…

1be750b

… use camelCase for metadata Signed-off-by: Mehmet TOSUN <mehmet.tosun@gmail.com>

CasperGN reviewed Dec 15, 2025

View reviewed changes

middt force-pushed the dapr-state-store-clickhouse branch from 45c198f to da6d1a0 Compare December 16, 2025 08:39

dapr-bot added 2 commits December 16, 2025 00:50

Merge branch 'main' into dapr-state-store-clickhouse

932d119

Merge branch 'main' into dapr-state-store-clickhouse

27ed098

	Copyright 2021 The Dapr Authors
	Copyright 2025 The Dapr Authors

Dapr state store clickhouse #3675

Are you sure you want to change the base?

Dapr state store clickhouse #3675

Conversation

middt commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation Details

Checklist

Testing Done

Additional Notes

Uh oh!

JoshVanL commented Mar 13, 2025

Uh oh!

middt commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshVanL commented Apr 9, 2025

Uh oh!

github-actions bot commented Nov 8, 2025

Uh oh!

github-actions bot commented Nov 15, 2025

Uh oh!

sicoyle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sicoyle commented Dec 9, 2025

Uh oh!

CasperGN commented Dec 15, 2025

Uh oh!

middt commented Dec 15, 2025

✅ Changes Made:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

middt commented Feb 23, 2025 •

edited

Loading

middt commented Mar 14, 2025 •

edited

Loading