Ctrlk
Hiero.orgCalendar
For the complete documentation index, see llms.txt. This page is also available as Markdown.

WRB CLI Runbook

Overview

This runbook documents the setup and operational workflows for the Wrapped Record Block (WRB) CLI tools. These tools are used to download, wrap, validate, and maintain block streams for Hiero networks (mainnet and testnet).

Audience: Operators running Block Stream validation, migration, or archival processes.

Prerequisites

Required Tools

# Install Java 25+
sudo apt-get update
sudo apt-get install openjdk-25-jdk

# Install zstd (required for decompression)
sudo apt-get install zstd

# Verify installations
java -version
zstd --version

Building the Shadow JAR

The shadow jar is self-contained and can be copied to your deployment server.

Optional: Command Aliases

For convenience, you can set up aliases to simplify command invocations:

Note: Commands in this runbook show the full java -jar syntax for clarity, but aliases can reduce typing for frequent operations.

Directory Structure

Recommended layout for operations:

Network Configuration

GCP Authentication (for GCS access)

Export GCP credentials before running any commands:

Network Selection

All commands accept --network <mainnet|testnet>:

  • Mainnet: --network mainnet (default if omitted)

  • Testnet: --network testnet

Core Workflows

1. Generate Address Book History

Purpose: Fetch address book changes from the Mirror Node API to enable block proof verification.

Command:

Output: wrappedBlocks/addressBookHistory.json contains all address book updates.

Notes:

  • Run this before starting wrapping or validation

  • Re-run periodically to capture new address book changes

Fallback: Generate from Binary File (For Networks Without Mirror Node Backups)

For networks like previewnet where Mirror Node CSV backups are not available, you can generate the address book history from a raw protobuf binary file (file 0.0.102 from the database).

Command:

Options:

How to obtain the bin file:

  • Export file 0.0.102 from the network's database

  • For previewnet: Contact the network operator for the latest address book export

  • The file contains the protobuf-serialized NodeAddressBook message

Output: Same format as generateAddressBook - a JSON file with address book history that can be used for block proof verification.

When to use:

  • Previewnet or custom networks without Mirror Node CSV exports

  • Emergency recovery when Mirror Node is unavailable

  • Testing with historical address book snapshots

2. Update Metadata Files

Metadata files are essential for mapping blocks to days and timestamps.

Generate Block Times and Day Blocks

Outputs:

  • metadata/block_times.bin - binary map of block numbers to consensus timestamps

  • metadata/day_blocks.json - JSON map of dates to block ranges

For bounded updates (e.g., stopping at a specific date):

Update Day Listings

What it does: Queries GCS for all record files per day and caches the listings locally.

Fix corrupt/missing day:

3. Download Block Data

Bulk Historical Download (download-days-v3)

Purpose: Download compressed record files for a specific date range.

Command:

Arguments:

  • --listing-dir: Path to day listings metadata

  • --downloaded-days-dir (or -d): Output directory

  • --threads (or -t): Parallelism (recommend 100-600 based on server capacity)

  • Date range: YYYY MM DD YYYY MM DD (start and end, inclusive)

Example - Single Day:

Monitoring Progress:

Resuming: The command automatically resumes from where it left off if interrupted.

Live Streaming Download (live-sequential)

Purpose: Continuously download, wrap, and validate blocks in near real-time.

Command:

Arguments:

  • -l: Listing directory

  • -o: Downloaded days output directory

  • --wrap-output-dir: Wrapped blocks output directory

  • --address-book: Path to address book history JSON

  • --start-date (optional): Start from specific date (format: YYYY-MM-DD)

With Start Date:

What it does:

  1. Polls GCS for new record files

  2. Downloads them to compressedDays/

  3. Wraps them into Block Stream format

  4. Writes to wrappedBlocks/

  5. Validates hashes inline

Monitoring:

Graceful Shutdown:

Legacy Live Download (download-live2)

Note: download-live2 is deprecated in favor of live-sequential. Use live-sequential for new deployments.

Command (for reference):

4. Wrap Record Files

Purpose: Convert compressed record files into the wrapped Block Stream format.

Command:

Arguments:

  • -i: Input directory (compressed days)

  • -o: Output directory (wrapped blocks)

  • -b: Block times binary file

  • -d: Day blocks JSON file

JVM Flags Explained:

  • -Xms20g -Xmx20g: Heap size (adjust based on available RAM)

  • -XX:+UseZGC: Low-latency GC (recommended for large heaps)

  • -XX:+AlwaysPreTouch: Pre-allocate heap pages (reduces runtime delays)

  • -XX:+UseNUMA: NUMA-aware allocation (if hardware supports it)

  • -XX:ConcGCThreads=12: GC thread count (tune to CPU cores)

Monitoring:

Resuming: Wrapping automatically resumes from the last completed block.

Output Structure:

Understanding Wrap Output Files

The wrapping process produces several auxiliary files in addition to the wrapped blocks:

jumpstart.bin

Purpose: Contains integrity verification data for Consensus Nodes performing WRB catch-up.

Contents:

  • Current block number

  • Block hash (SHA-384)

  • Consensus timestamp hash (SHA-384)

  • Output items tree root hash (SHA-384)

  • Streaming hasher state (for incremental hash verification)

When created: Generated during blocks wrap and days live-sequential operations.

Consumed by: Consensus Nodes during fast catch-up to verify wrapped block integrity.

Management:

Delete jumpstart.bin when:

  • Regenerating wrapped blocks for a date range (to avoid stale state)

  • live-sequential reports validation mismatches

  • Switching networks or starting fresh wrapping operations

  • The file is older than the wrapped blocks it references

Example - Regenerating after corruption:

Note: The jumpstart file is automatically regenerated during wrapping. Deleting it is safe and often necessary when re-wrapping existing block ranges.

tss-enablement.bin

Purpose: Binary TSS (Threshold Signature Scheme) parameters file for Block Node verification plugins.

Contents:

  • Raw protobuf binary of the most recent LedgerIdPublication transaction

  • TSS public key material

  • Ledger ID

  • Signature share thresholds

When created: Automatically written when the first LedgerIdPublication transaction is encountered during wrapping or validation.

Consumed by: Block Node's VerificationServicePlugin for TSS-based block proof verification.

Management:

The file is updated automatically whenever a new TSS publication is detected. You typically do not need to manually manage this file unless:

  • Switching networks (TSS parameters differ per network)

  • Regenerating blocks from scratch (delete to allow fresh detection)

Example output during wrapping:

tssPublicationHistory.json

Purpose: JSON checkpoint file tracking all TSS publications discovered during wrapping.

Contents:

  • Block number and timestamp of each TSS publication

  • Ledger ID changes over time

  • TSS public key transitions

  • Validity ranges (start/end blocks)

When created: Saved periodically during wrapping operations as part of validation checkpoints.

Consumed by: Resume logic for blocks wrap and live-sequential to maintain TSS state across restarts.

Management:

This file is managed automatically by the validation checkpoint system. It is safe to delete when:

  • Starting fresh wrapping operations

  • Switching networks

  • The corresponding tss-enablement.bin has been deleted

Example structure:

5. Validate Wrapped Blocks

Purpose: Verify the integrity of wrapped blocks (hash chain, merkle trees, HBAR supply, balances).

Full Validation (Mainnet):

Skip Supply Validation (for testnet or known supply issues):

Verbose Mode (detailed output per block):

Available Flags:

Flag
Purpose

--skip-supply

Skip HBAR supply validation

--skip-signatures

Skip block proof signature validation

--validate-balances=false

Disable balance tracking validation

--verbose

Print detailed per-block output

--threads <N>

Set parallel validation threads (default: CPU cores)

--prefetch <N>

Number of blocks to prefetch (default: 10)

--no-resume

Start from block 0 (ignore saved checkpoint)

Checkpoint Behavior:

  • Validation saves checkpoints periodically to wrappedBlocks/.validation_checkpoint

  • Resume automatically continues from the last checkpoint

  • Graceful shutdown (kill) saves checkpoint; kill -9 does not

Interpreting Output:

  • Speed multiplier: 2.1x means validation is 2.1x faster than block production

  • Timing breakdown: Shows time spent on each validation phase (hashing, signatures, balances)

Known Issues:

Testnet Supply Validation Failure (Block 7,557,270):

  • A SCHEDULESIGN transaction with unbalanced transfer list causes supply mismatch

  • Workaround: Use --skip-supply for testnet validation

  • Root cause: Testnet Consensus Node bug from HAPI v0.52.0 (August 2024)

  • Reference: Testnet Mirror API

TSS Enablement Detection:

During validation, the CLI automatically detects and extracts TSS (Threshold Signature Scheme) enablement data from LedgerIdPublication transactions. When detected, two files are written to the wrapped blocks directory:

Output Files:

File
Format
Purpose

tss-enablement.bin

Protobuf binary

TssData in protobuf format for archival

tss-bootstrap-roster.json

JSON

TssData in JSON format matching Block Node's ApplicationStateConfig

Example JSON Output (tss-bootstrap-roster.json):

Usage:

  • The JSON file can be directly consumed by the Block Node's ApplicationStateConfig

  • Both files are updated automatically on each TSS publication detection

  • Validation output shows: TSS publication at block <N>: <description>

6. Repair Corrupt or Missing Files

Purpose: Re-download individual corrupt record files.

Command:

What it does:

  • Scans compressedDays/ for corrupt or incomplete .zip files

  • Re-downloads them from GCS using the day listings

  • Verifies file integrity after download

Use case: When download-days-v3 or wrap reports file corruption errors.

Operational Notes

Monitoring Progress

Active Processes:

Log Tailing:

Disk Usage:

Block Count:

Disk Space Requirements

Recommendations:

  • Use ZFS with compression (ratio ~1.5x for wrapped blocks)

  • Separate volumes for compressed days and wrapped blocks

  • Monitor disk usage during bulk downloads

Graceful Shutdown

SIGINT (recommended):

  • Allows checkpoint save

  • Cleanly closes file handles

  • Safe for resume

SIGKILL (avoid):

  • Loses checkpoint state

  • May corrupt in-progress writes

  • Use only if process is unresponsive

Running with nohup

Pattern:

  • nohup: Prevents termination on terminal disconnect

  • > output.log: Redirect stdout to log file

  • 2>&1: Redirect stderr to stdout (same log)

  • &: Run in background

Check Job:

Bring to Foreground (if needed):

Troubleshooting

"Unknown file type in GCS bucket"

Symptom: Download fails with unknown file extension error.

Cause: GCS contains non-record files (e.g., .tmp, .lock).

Solution: Re-run days updateDayListings for the affected day to refresh the file list.

"HBAR supply mismatch"

Symptom: Validation fails at specific block with supply error.

Cause: Consensus Node bug or testnet-specific issue.

Solution: Use --skip-supply flag. Report issue if on mainnet.

"Block proof signature verification failed"

Symptom: Signature validation fails.

Cause: Missing or outdated address book history.

Solution:

  1. Re-generate address book: mirror generateAddressBook

  2. Use --skip-signatures as temporary workaround

"Out of memory" during validation

Symptom: JVM crashes with OutOfMemoryError.

Solution:

  1. Increase heap: -Xms32g -Xmx32g

  2. Reduce prefetch: --prefetch 5

  3. Reduce threads: --threads 8

Download stalls or times out

Symptom: download-days-v3 hangs or reports timeout.

Solution:

  1. Reduce threads: --threads 100 (instead of 600)

  2. Check network connectivity to GCS

  3. Verify GCP credentials are valid

Validation mismatches with stale jumpstart.bin

Symptom: live-sequential or validation reports hash mismatches or integrity errors.

Cause: Stale jumpstart.bin file from previous wrapping operations.

Solution:

When this happens:

  • After regenerating blocks for a date range that overlaps with existing wrapped blocks

  • When switching between networks (mainnet/testnet)

  • If wrapping was interrupted and restarted from a different starting block

Quick Reference

Common Command Patterns

Mainnet Full Pipeline:

Testnet Full Pipeline (with --skip-supply):

Appendix: Real-World Examples

Mainnet Production Setup

Hardware:

  • 32-core server

  • 128GB RAM

  • 20TB ZFS RAID10 SSD array

Commands:

Testnet CI/CD Validation

Purpose: Nightly validation of testnet wrapped blocks.

PreviousPreparing for WRB Cutover

Last updated