Split state persistent on parts

[SmaGMan]

RATIONALE

Support local state data sharding on parts - shard accounts cells tree is split into parts by shards at the configured split_depth. The top of the state tree is stored in the main cells db, and parts subtrees are stored each in separate physical databases that can be placed on separate disks.

---

Pull Request Checklist

NODE CONFIGURATION MODEL CHANGES

[Yes]

Added core_storage.state_parts

...
"core_storage": {
  ...
  "state_parts": {
    "split_depth": N, // depth of the state split on 2^N parts
    "part_dirs" : {
      // key - hex representation of part shard prefix; value - path to the part database
      "a000000000000000": "path/to/cells-part-a000000000000000",
      ...
    }
  }
  ...
}
...

• when split_depth: 0 - no parts used;
• we can set a custom database path, even only for one part, so we can move only one database to a separate disk if required;
• if the path to part database is not specified, then the relative path used "cells-parts/cells-part-{shard prefix hex}".

Default value is state_parts: null that means no parts configured.

BLOCKCHAIN CONFIGURATION MODEL CHANGES

[None]

---

COMPATIBILITY

Affected features:

• [State]
• [Persistent State]
• [Storage. Blocks]
• [Storage. States]
• [Rpc]

Fully compatible.

State will be saved with parts if they are specified in config. If state was saved with parts it will be read with parts. If it was saved without parts (e.g. before update) it will be read without.

Parts map {key - cell hash: value - shard prefix} will be saved to CellsDB.shard_states right after root cell hash. If no parts used nothing will be added to ShardStates value. So existing values in ShardStates table will be treated as "no parts used".

A new flag HAS_STATE_PARTS = 1 << 13 added to the BlockHandle bit flags. It means that no parts were used / or all required state parts were successfully stored in separate storages. Now BlockHandle.has_state() returns true only when both new flag and old one HAS_STATE_MAIN = 1 << 3 are set. The migration script (0.0.4 -> 0.0.5) set HAS_STATE_PARTS for all existing block handles.

BUT parts configuration changes (e.g. from 4 to 8 partitions, or from 8 to 2 or 0) are not auto compatible. Will be implemented in a separate task.

Persistent state files will be split on main file and parts if persistent was stored when state split on parts configured. E.g.:

• main file: {block_id}.boc
• parts files: {block_id}_part_{shard prefix hex}.boc

Persistent part file has additional meta after header at the offset 6. It contains shard prefix (8 bytes) and part root hash (32 bytes).

Persistent main file has additional meta when stored with parts:

• it will contain FLAG_HAS_PARTS (0b1000) in the header
• it will contain split_depth value (1 byte) right after header at the offset 6 if it contains flag FLAG_HAS_PARTS
• parts subtrees will be replaced with corresponding pruned branches

Previous persistent state file format is fully backward compatible.

Manual compatibility tests were passed:

• set core_storage.state_parts = null or remove param from config
• build last master version
• gen local network

just gen_network 1 --force

• run node

 just node 1

• run 20k transfers test

 ./transfers-20k.sh

• stop node
• build feat/split-state-storage barch version
• run node without reset

 just node 1

• see successful core db migration to 0.0.5 in logs
• continue 20k transfers test, ensure all is going well

 ./transfers-20k.sh --continue

• stop node
• set up 4 parts in .temp/config1.json

...
  "state_parts" : {
    "split_depth": 2
  }
...

• run node without reset

 just node 1

• continue 20k transfers test, ensure all is going well

 ./transfers-20k.sh --continue

• stop node
• move some parts databases

 mkdir .temp/db1/cells-parts-moved
 mv .temp/db1/cells-parts/cells-part-a000000000000000 .temp/db1/cells-parts-moved/
 mv .temp/db1/cells-parts/cells-part-6000000000000000 .temp/db1/cells-parts-moved/

• set up paths to moved databases in .temp/config1.json

...
  "state_parts" : {
    "split_depth": 2,
	"part_dirs": {
	  "a000000000000000": "/workspace/tycho/.temp/db1/cells-parts-moved/cells-part-a000000000000000",
      "6000000000000000": "cells-parts-moved/cells-part-6000000000000000"
	}
  }
...

• run node without reset

 just node 1

• continue 20k transfers test, ensure all is going well

 ./transfers-20k.sh --continue

Manual persistent state test 1:

• set 4 parts in the node config.json

...
  "state_parts" : {
    "split_depth": 2
  }
...

• gen local network of 4 nodes

just gen_network 4 --force

• run 3 nodes with hack when every key block is persistent

RUSTFLAGS="--cfg tycho_unstable" HACK_EACH_KEY_BLOCK_IS_PERSISTENT=1 just node 1
RUSTFLAGS="--cfg tycho_unstable" HACK_EACH_KEY_BLOCK_IS_PERSISTENT=1 just node 2
RUSTFLAGS="--cfg tycho_unstable" HACK_EACH_KEY_BLOCK_IS_PERSISTENT=1 just node 3

• run 20k transfers test to deploy 20k wallets

 ./transfers-20k.sh

• stop transfers and change bc config params to force a key block

target/debug/tycho tool bc set-param --rpc http://localhost:8001 --key <key> \
22 \
'{
  "bytes": {
    "underload": 1000,
    "soft_limit": 5000,
    "hard_limit": 6000
  },
  "gas": {
    "underload": 900000,
    "soft_limit": 15000000,
    "hard_limit": 20000000
  },
  "lt_delta": {
    "underload": 1000,
    "soft_limit": 10000,
    "hard_limit": 20000
  }
}'

• check that persistent state was created with parts

ls -lah .temp/db1/files/states

• run node 4 without zerostate (make a separate run-node4.sh script where --import-zerostate arg is omitted)

RUSTFLAGS="--cfg tycho_unstable" HACK_EACH_KEY_BLOCK_IS_PERSISTENT=1 ./scripts/run-node4.sh

• check that node 4 downloaded zerostate and persistent state from other nodes, successfully stored them and joined further blocks collation

If nodes 1-3 where configured without state split on parts and persistent state was created without parts it will be correctly downloaded and stored on the node 4 (that configured with 4 parts). We will have a single persistent state file but state will be split stored into 4 different storages.

SPECIAL DEPLOYMENT ACTIONS

[Not Required]

Without additional changes in the node config it works with a single part without split.

---

PERFORMANCE IMPACT

[Expected impact]

• Better perfomance of non-zero states (~20-30%)
  • master: degradation on 20k transfers from empty to 30kk state: from ~35k tps to ~15-20k tps

• 4 local parts: degradation on 20k transfers from empty to 30kk state: from ~35k tps to ~20-30k tps

• No states GC lag growth
• Faster state store

---

TESTS

Unit Tests

• test_preload_persistent_states
• test_store_shard_state_from_file

Network Tests

[No coverage]

Manual Tests

Performance testing:

• 20k transfers
• 30k transfers
• deploy 30kk accounts
• 20k transfers
• 30k transfers

(metrics are in the PERFORMANCE IMPACT block)

[github-actions]

🧪 Network Tests

To run network tests for this PR, use:

gh workflow run network-tests.yml -f pr_number=983

Available test options:

• Run all tests: gh workflow run network-tests.yml -f pr_number=983
• Run specific test: gh workflow run network-tests.yml -f pr_number=983 -f test_selection=ping-pong

Test types: destroyable, ping-pong, one-to-many-internal-messages, fq-deploy, nft-index, persistent-sync

Results will be posted as workflow runs in the Actions tab.

[github-actions]

❌ Python formatting check failed in CI.

Please run just fmt_py locally and push the updated files.

[codecov]

Codecov Report

❌ Patch coverage is 58.13464% with 1194 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.83%. Comparing base (4e0f1cc) to head (af375e8).

Files with missing lines	Patch %	Lines
core/src/storage/shard_state/mod.rs	45.26%	280 Missing and 15 partials ⚠️
core/src/storage/persistent_state/mod.rs	54.71%	157 Missing and 40 partials ⚠️
...storage/persistent_state/parts/impls/local_impl.rs	37.75%	146 Missing and 4 partials ⚠️
...e/src/storage/persistent_state/descriptor_cache.rs	62.62%	92 Missing and 16 partials ⚠️
core/src/storage/shard_state/cell_storage.rs	56.54%	87 Missing and 6 partials ⚠️
core/src/storage/db.rs	46.15%	53 Missing and 3 partials ⚠️
core/src/storage/persistent_state/tests.rs	80.68%	7 Missing and 44 partials ⚠️
core/src/blockchain_rpc/client.rs	30.18%	36 Missing and 1 partial ⚠️
core/src/storage/shard_state/store_state_raw.rs	76.51%	20 Missing and 15 partials ⚠️
core/src/block_strider/starter/cold_boot.rs	0.00%	32 Missing ⚠️
... and 18 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #983      +/-   ##
==========================================
+ Coverage   46.30%   47.83%   +1.52%     
==========================================
  Files         357      361       +4     
  Lines       62235    64416    +2181     
  Branches    62235    64416    +2181     
==========================================
+ Hits        28820    30811    +1991     
+ Misses      31914    31911       -3     
- Partials     1501     1694     +193     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.