Mutable.ai logoAuto Wiki by Mutable.ai

bitcoin

Auto-generated from bitcoin/bitcoin by Mutable.ai Auto Wiki

bitcoin
GitHub Repository
Developerbitcoin
Written inC++
Stars73k
Watchers 4.0k
Created2010-12-19
Last updated2023-12-25
LicenseMIT
Homepagebitcoincore.org/en/download
Repositorybitcoin/bitcoin
Auto Wiki
Generated at2023-12-25
Generated fromCommit 4b1196
Version0.0.4

Bitcoin Core is an open source software project that maintains and releases Bitcoin client software called Bitcoin Core. The code is primarily written in C++ and contains everything needed to parse, validate, store and relay bitcoin transactions and blocks as well as wallet functionality for managing keys and transactions.

Some of the most important components and algorithms in Bitcoin Core include:

  • The Chainstate class defined in chain.h which represents the entire UTXO database and handles all chainstate-related functionality like validating new blocks, connecting and disconnecting blocks, and efficiently querying UTXO data.

  • The peer-to-peer networking layer defined in files like net.h which handles establishing connections to peers, propagating transactions and blocks, responding to requests for blockchain data, and implementing the low-level Bitcoin protocol. This uses inbound and outbound connections and message queues to efficiently relay data.

  • The script module defined across files like script.h which parses, represents, evaluates and validates the transaction scripts used in Bitcoin. Key algorithms include the EvalScript function which interprets scripts by executing opcodes.

  • The CWallet class defined in wallet.h which represents the wallet and handles key management, transaction building using coin selection, fee estimation, encryption and decryption, persistence, and integration with GUI wallets.

  • The LevelDB-based transaction index defined in txdb.h which maintains an index of transactions by txid to allow fast lookup of transactions without needing to scan blocks. This uses a key-value database to maintain the mapping.

  • The CBlock and CTransaction classes defined in primitives/block.h and primitives/transaction.h which represent Bitcoin blocks and transactions using data structures that can be serialized for storage and transmission over the network.

  • The secp256k1 library located in src/secp256k1 which provides optimized implementations of cryptographic algorithms used in Bitcoin, including ECDSA signing/verification, public/private key operations, and hash functions.

  • The consensus validation logic defined across files like consensus/tx_verify.cpp which contain the business logic for validating blocks and transactions against Bitcoin consensus rules. This is critical to maintaining agreement across nodes.

The codebase is structured into logical components and layers, contains extensive tests and benchmarks, and leverages various optimization techniques to efficiently handle Bitcoin's data structures and algorithms. The modular architecture, focus on security, and active development community make Bitcoin Core a robust implementation of Bitcoin.

Consensus Rules and Validation

References: src/consensus

The Bitcoin codebase implements consensus rules and validation through rigorous checking of transactions and blocks. Key components work together to enforce the rules of the Bitcoin protocol.

The …/consensus directory contains core consensus logic and definitions. It establishes common types like CAmount for representing monetary values in …/amount.h. The Params struct in …/params.h defines network parameters for mainnet, testnet and regtest.

Merkle roots are computed through functions in …/merkle.cpp. ComputeMerkleRoot() calculates transaction roots, handling odd-sized lists. BlockMerkleRoot() and BlockWitnessMerkleRoot() leverage it to compute roots for blocks.

The ValidationState template in …/validation.h tracks validation results. Specialized subclasses are used for transactions and blocks. Weight calculation functions allow accurately determining weights based only on serialization sizes.

Transaction validation occurs across several files. …/tx_check.cpp contains the CheckTransaction function for basic checks. …/tx_verify.cpp implements locktime and sequence number logic through functions like IsFinalTx(), CalculateSequenceLocks() and SequenceLocks(). It also contains signature counting functions.

The CheckTransaction function in …/tx_check.h allows stateless validation, while …/tx_verify.cpp validates during consensus. Together these files enforce critical rules for transaction validity.

Block validation logic is also implemented across multiple files to thoroughly check blocks comply with consensus rules before accepting them in the best chain.

Consensus Parameters

References: src/consensus/params.h

The file …/params.h defines the key network parameters that influence the consensus rules for the Bitcoin blockchain. This includes definitions for:

  • Enumerations like Consensus::DeploymentPos that specify buried deployments of new consensus rules at hardcoded block heights.

  • The BIP9Deployment struct describing version bits parameters for individual consensus rule changes.

  • The critical Params struct containing consensus parameters for the main chain like:

    • The genesis block hash

    • Block subsidy halving interval

    • Block heights for activating consensus rules such as BIP34, BIP65, BIP66, and others

    • Proof-of-work parameters like the pow limit and target timespan/spacing

    • The minimum amount of work the best chain must have

    • The default block a node will assume is valid

The Params struct stores these consensus rules and constants that nodes use to validate blocks. It contains the vDeployments vector holding the version bits deployment parameters for individual soft forks. Buried deployment heights can be retrieved via the DeploymentHeight() function.

Transaction Validation

References: src/consensus/tx_verify.cpp, src/consensus/tx_verify.h, src/consensus/validation.h

The core files for transaction validation in Bitcoin Core are …/tx_verify.cpp and …/tx_verify.h. These files contain the implementation of the Consensus::CheckTxInputs function, which validates that all transaction inputs are not double spends and have valid amounts.

The Consensus::CheckTxInputs function checks each transaction input by looking up the referenced output in the UTXO set (CCoinsViewCache). It validates that the input is not already spent, and that the amount being spent does not exceed the amount of the referenced output.

Other important validation logic includes checking that transactions respect locktimes and sequence numbers. The IsFinalTx function determines if a transaction's locktime has passed to know if it is valid to include in a block. The CalculateSequenceLocks, EvaluateSequenceLocks, and SequenceLocks functions together implement validation of input sequence numbers based on BIP68.

Transaction validation also counts signature operations using the GetLegacySigOpCount and GetP2SHSigOpCount functions. These count sigops for non-P2SH and P2SH outputs respectively by looking up previous transactions. GetTransactionSigOpCost combines these to give an overall signature operation cost metric for a transaction.

This signature counting is important because it is used to validate transactions do not exceed the maximum allowed signature operations in a block. Validating signature operations ensures transactions respect consensus limits on processing requirements.

Block Validation

References: src/consensus/validation.h, src/consensus/tx_check.cpp, src/consensus/tx_check.h

Block validation is performed by the CheckBlock function defined in …/validation.h. This function validates a block against the network consensus rules by checking various attributes of the block header and validating each transaction in the block.

The CheckBlock function takes a CBlock object and BlockValidationState object as parameters. It first performs validation checks on the block header such as checking the block hash against the expected proof-of-work, validating the block version and timestamp.

Transactions are then validated using the CheckTransaction function defined in …/tx_check.cpp. This function validates each transaction and ensures no double spends or invalid outputs occur. Transaction inputs are also script validated.

If any part of block header validation or transaction validation fails, an error is recorded in the BlockValidationState object. This tracks any errors encountered and block validation result. After all checks pass, CheckBlock returns true.

Merkle Trees

References: src/consensus/merkle.cpp, src/consensus/merkle.h

Merkle trees are used in Bitcoin to efficiently validate transactions in blocks. The …/merkle.cpp file contains the core ComputeMerkleRoot() function which calculates the Merkle root of a vector of transaction hashes by iteratively hashing pairs of elements moving up the tree. It handles odd sized lists by duplicating the last hash.

The BlockMerkleRoot() and BlockWitnessMerkleRoot() functions leverage ComputeMerkleRoot() to compute the specific Merkle roots needed for blocks. BlockMerkleRoot() extracts the transaction hashes from a block and passes them to ComputeMerkleRoot() to get the regular block root. BlockWitnessMerkleRoot() similarly builds a vector of hashes, but sets the coinbase hash to empty since witness data is not included in coinbase transactions. It then calls ComputeMerkleRoot() to obtain the witness root.

ComputeMerkleRoot() takes the vector of hashes and first checks for any identical duplicate pairs, setting a mutation flag if found. It then iteratively hashes pairs of elements using SHA256D64() to combine them as it resizes the vector after each round. This continues until a single root hash is computed. The functions in …/merkle.h define the same algorithms, with ComputeMerkleRoot computing the root over a generic vector and the other functions specialized for block data.

Consensus Types

References: src/consensus/amount.h, src/consensus/validation.h

The file …/amount.h defines types and constants used for representing monetary amounts in Bitcoin. The type CAmount is defined as a 64-bit signed integer to store amounts in satoshis, allowing amounts between -9.223 quintillion satoshis and 9.223 quintillion satoshis to be represented. The constant COIN is defined as 100,000,000 satoshis to easily convert between BTC and satoshis. MAX_MONEY is defined as 21 million BTC to represent the maximum possible amount of BTC that can exist as defined by the protocol.

The inline function MoneyRange takes a CAmount and returns true if it is between 0 and MAX_MONEY. This check validates amounts are in the acceptable range before use and is important for consensus critical security.

The file …/validation.h defines enums for possible validation results of transactions (TxValidationResult) and blocks (BlockValidationResult). The template class ValidationState captures validation information and status, with specialized subclasses TxValidationState and BlockValidationState used during transaction and block validation respectively.

Utility functions directly implement the BIP 141 weight formula, calculating weights based only on serialization sizes. The GetWitnessCommitmentIndex function searches a block's coinbase transaction for the output containing the witness commitment hash.

P2P Networking

References: bitcoin

The Bitcoin P2P network allows nodes to communicate transaction and block data to each other over a messaging protocol. This section will discuss how Bitcoin Core implements peer-to-peer networking functionality.

The CNode class defined in …/net.h represents an individual peer or node connected to the local Bitcoin node. CNode objects store metadata about the peer like its address, services, and user agent. They also track statistics like bytes received and sent.

The CConnman class defined in …/net.h manages the overall P2P network connections and relay process. It maintains a list of all connected CNode peers. Methods like OpenNetworkConnection() handle connecting to new peers, and ProcessMessages() relays blocks and transactions between nodes.

Messages are serialized and deserialized using classes defined in …/protocol.h. For example, the CMessageHeader class handles serialization of message headers. Messages are sent and received via queues on each CNode.

The msg_version and msg_verack messages defined in …/protocol.h handle the initial version handshake between peers. The local node sends msg_version to advertise its capabilities and services. It expects msg_verack in response to acknowledge the connection.

The msg_inv and msg_getdata messages allow peers to asynchronously exchange inventory vectors of transaction and block hashes. Nodes send msg_inv to broadcast available data, and receive msg_getdata requests to then send the full transaction and block data.

Blockchain and UTXO Storage

References: src/index

The Bitcoin Core codebase includes several components for efficiently storing, indexing, and querying blockchain data like transactions and blocks. Key components include databases for storing raw block and transaction data on disk, and secondary indexes to enable fast lookups without linear scanning.

The …/index directory contains several secondary indexes to further optimize access to blockchain data. The TxIndex in …/txindex.cpp indexes transactions by txid, storing their disk positions, enabling fast retrieval without scanning blocks. The BlockFilterIndex in …/blockfilterindex.cpp stores and indexes block filter data, allowing efficient lookups by height or hash. The CoinStatsIndex in …/coinstatsindex.cpp maintains statistics on the UTXO set like output counts and amounts to facilitate metrics and analysis.

All indexes inherit functionality from the BaseIndex class in …/base.cpp, which handles initializing from the database, syncing new blocks via callbacks, and committing changes. Indexes implement methods like CustomAppend to update when blocks connect, and CustomCommit to flush data. The DB subclasses abstract database interactions.

Block Storage

References: bitcoin

Block data is stored on disk in individual files. The CBlock class defined in …/block.h represents a full Bitcoin block, containing a vector of transactions and other metadata.

The BlockManager class defined in …/blockmanager_opts.h handles block validation and storage. One of its responsibilities is persisting block data to disk. It interacts with the CoinsView interface to retrieve and store block data.

The CoinsView interface abstracts the underlying database. When a new block connects, the BlockManager calls methods on CoinsView like GetBlock() to retrieve the block's raw transactions from disk. It then validates the block before accepting it into the chain.

For storing, the BlockManager calls methods on CoinsView like SetBestBlock() once a block is fully validated and added to the chain. This persists the new chain state, including writing the block's raw transaction data to disk if it has not already been written. Data is stored sequentially by height in individual files. This file-based storage allows efficient block retrieval and validation during initial block download.

Chain State

References: src/chain.h, src/chain.cpp, src/coins.h, src/coins.cpp

The CCoinsViewCache class is the primary interface for maintaining and querying the UTXO set and chain state in memory. It implements an in-memory cache on top of a backing CCoinsView, such as CCoinsViewDB which retrieves UTXOs from the database.

CCoinsViewCache stores cached UTXOs, represented by Coin objects, in a CCoinsMap. Methods like FetchCoin() retrieve coins from the cache, while AddCoin() and SpendCoin() are used to update the cache during transaction validation. The cache tracks metadata for each coin like whether it is "dirty" (changed and needs flushing) or "fresh" (newly added).

CCoinsViewCache handles flushing any dirty coins back to the underlying view through BatchWrite(). This synchronizes the in-memory state with the backing database. The cache also tracks memory usage and can be flushed via Flush(). Memory management is important since the UTXO set can grow very large.

Other key functionality includes utilities for adding/removing coins when processing transactions. The AccessByTxid() method retrieves coins involved in a transaction. Coins can also be retrieved and iterated over range queries by output point hashes.

Block Indexing

References: src/txdb.h, src/txdb.cpp

The CCoinsViewDB class provides block indexing by height and hash for fast lookup through its underlying LevelDB database. CCoinsViewDB stores the block hash of the best block in the database, which allows it to retrieve the current tip of the chain quickly on startup.

The database maps block hashes to block heights through the CBlockIndex structures, which are stored in the LevelDB blockindex bucket. To retrieve a block by hash, CCoinsViewDB looks up the corresponding CBlockIndex in this bucket, which contains the block height. It can also iterate through all CBlockIndex structures in height order to retrieve the main chain.

The CBlockIndex structures contain metadata about each block like its height and hash. They are stored in the blockindex bucket, with the block hash as the key. The LoadBlockIndex function builds the initial CBlockIndex structures from the block files.

The CCoinsViewDB class provides methods like GetBestBlock and GetHeadBlocks that utilize this block index to retrieve block headers without scanning the entire block chain from disk. GetBestBlock looks up the best block hash stored in the database and returns its header. GetHeadBlocks iterates through the indexed blocks in height order to return the headers of the main chain.

Transaction Indexing

References: src/index/txindex.h, src/index/txindex.cpp

The TxIndex class provides an interface for indexing transactions by txid (transaction ID hash) in the Bitcoin Core wallet, enabling fast retrieval of transactions without linearly scanning all blocks. The TxIndex class inherits from BaseIndex and is responsible for maintaining the transaction index database in LevelDB.

When new blocks are connected to the chain, TxIndex's CustomAppend method is called. This method uses the interfaces::Chain interface to retrieve the transactions included in the newly appended block. It then writes each transaction's hash and metadata like the block height and position to the underlying TxIndex::DB database.

The TxIndex::DB class implements the low-level storage and retrieval of transaction positions from the LevelDB database. It subclasses BaseIndex::DB and stores each transaction's position indexed by the hash with a key of DB_TXINDEX. The ReadTxPos method provides the interface to lookup a transaction's position by its hash. WriteTxs allows batch writing of multiple positions to improve performance during block connects.

To retrieve a transaction, TxIndex's FindTx method first calls TxIndex::DB's ReadTxPos to obtain the transaction's position. It then opens the corresponding block file and parses the transaction at that location. This allows looking up and retrieving any transaction contained in the blockchain by only its hash. The LevelDB backend enables fast random access reads and writes of the unordered transaction position data.

UTXO Database

References: src/coins.h, src/coins.cpp

The CCoinsViewCache efficiently stores and indexes the set of unspent transaction outputs (UTXOs) for fast lookup. It uses a CCoinsMap implemented as an unordered map to store UTXOs, indexed by their outpoint (COutPoint). This map provides quick lookup of UTXOs by transaction and output index.

The CCoinsViewCache caches UTXOs in memory on top of a backing CCoinsView store, such as CCoinsViewDB which retrieves UTXOs from the database. This in-memory cache improves performance by avoiding repeated database lookups. Methods like AddCoin() and SpendCoin() are used to update the cache during transaction validation. FetchCoin() retrieves coins from the cache or underlying view.

CCoinsViewCache tracks cached coins as "dirty" if modified, to know which changes need flushing to the backing store. On shutdown or periodically it calls BatchWrite() to flush dirty coins en masse for high performance. AccessByTxid() can retrieve all UTXOs associated with a transaction from the cache. Memory usage of the cache is tracked and it can be flushed or synced via Flush() and Sync().

Block Filter Index

References: src/index/blockfilterindex.h, src/index/blockfilterindex.cpp

The BlockFilterIndex class handles indexing and querying block filter data. It subclasses BaseIndex to inherit functionality for interacting with the chain and LevelDB database. Block filters are stored directly to disk in flat files that are managed by the FlatFileSeq class. This class tracks the current file and position as filters are appended.

The BlockFilterIndex indexes filters in LevelDB by block height and hash. This allows filters and headers to be looked up quickly by block criteria. The DBVal struct stores the associated data returned from lookups, such as the filter hash, header, and file position. Queries use DBHeightKey and DBHashKey to search by height or hash respectively.

When a new block is connected, the CustomAppend method writes its filter to disk and inserts it into the height index. The CustomCommit method flushes any unwritten files to storage. During reorganizations, CustomRewind facilitates copying filter data to the new chain. Lookup methods like LookupFilter retrieve filters from disk using the indexes.

A global map g_filter_indexes manages multiple BlockFilterIndex instances - one per supported filter type. Functions like GetBlockFilterIndex and InitBlockFilterIndex initialize and provide access to these indexes. This separation prevents collisions between filter formats.

Coin Statistics Index

References: src/index/coinstatsindex.h, src/index/coinstatsindex.cpp

The CoinStatsIndex class maintains running totals and statistics about the UTXO set such as total amounts, output and transaction counts, amounts grouped by category like coinbase outputs. It inherits from the BaseIndex class which defines the indexing API.

When a new block is connected via CustomAppend(), statistics like total amounts and outputs are recalculated. If a block is disconnected during a reorg, CustomRewind() is called to reverse these calculations. Stored statistics can be looked up via LookUpStats() using a block height or hash.

The CoinStatsIndex constructor initializes the underlying RocksDB database via the m_db member. As blocks are connected or disconnected, running totals for the statistics are maintained in memory. When the chain tip advances, updates are flushed to disk by CustomCommit() so the on-disk state is consistent with memory.

Several structs like DBVal are used to store and retrieve values from the indexes. DBHeightKey and DBHashKey define the keys used to index values by block height or hash respectively. Utility functions such as CopyHeightIndexToHashIndex() synchronize data between the two indexes.

Global variables track statistics like total amounts, while g_coin_stats_index is the singleton index instance that allows any part of the code to query statistics via LookUpStats().

Wallet Functionality

References: src/wallet, src/policy, src/rpc

The core wallet functionality implemented in Bitcoin Core involves managing keys, generating addresses, constructing transactions, estimating appropriate fees, and interfacing with the network and storage systems. This functionality is split across several components that work together to provide the wallet capabilities.

The CWallet class is the central class representing a loaded wallet. It tracks transactions, addresses, scripts, and balances in memory using maps like mapWallet and mapAddressBook. CWallet provides the core transaction creation logic by selecting UTXOs, building transactions, signing inputs, and broadcasting. It handles blockchain notifications to update the wallet state as new blocks connect.

ScriptPubKeyMan subclasses like LegacyScriptPubKeyMan and DescriptorScriptPubKeyMan manage keys, scripts, and addresses for different wallet types. They generate new addresses via methods like GetNewDestination() and sign transactions by traversing keys. CWallet stores a ScriptPubKeyMan to delegate key management duties.

Transaction fees are estimated using the CBlockPolicyEstimator class, which tracks historical fee data over different time horizons. Functions like estimateSmartFee() leverage this data to provide fee estimates. The CFeeRate class represents fee rates used throughout for calculations.

The wallet databases are represented by WalletDatabase subclasses for different backends like BerkeleyDatabase and SQLiteDatabase. These handle opening/closing databases, performing transactions via DatabaseBatch, and providing cursors for iteration.

The wallet RPC interface is implemented in …/rpc. Classes like CWallet are used to retrieve data, while JSON-RPC commands are registered via CRPCTable and implemented by calling the appropriate wallet methods.

Key Management

References: src/wallet/scriptpubkeyman.h

The LegacyScriptPubKeyMan and DescriptorScriptPubKeyMan classes are responsible for managing keys, addresses, and scripts for wallets. LegacyScriptPubKeyMan handles non-HD wallets by storing private keys in mapKeyMetadata, encrypted private keys in mapCryptedKeys, and watching-only scripts in setWatchOnly. It manages the keypool via setInternalKeyPool and setExternalKeyPool, generating new keys from the CHDChain model. To get a new address, it calls GetKeyFromPool() or ReserveKeyFromKeyPool(). TopUp() refills the keypool.

DescriptorScriptPubKeyMan handles descriptor-based wallets by caching descriptor information like scripts and pubkeys in m_map_script_pub_keys and m_map_pubkeys. To get a new address, it expands the descriptor using GetNewDestination(). Private keys are stored in m_map_keys and encrypted in m_map_crypted_keys. It generates FlatSigningProviders lazily via GetSigningProvider() to avoid regenerating them each time. TopUp() refills the descriptor cache.

The CKeyPool struct represents a keypool entry, storing the pubkey, internal/external flag, and metadata. WalletStorage defines common wallet storage functionality accessed by both LegacyScriptPubKeyMan and DescriptorScriptPubKeyMan.

LegacyScriptPubKeyMan manages private keys by storing them encrypted or unencrypted in mapKeyMetadata and mapCryptedKeys. It differentiates between internal and external keys by storing them in setInternalKeyPool and setExternalKeyPool. To generate new keys, it uses the CHDChain model and calls functions like GetKeyFromPool() and ReserveKeyFromKeyPool(). It signs transactions by providing its private keys to the SigningProvider interface.

DescriptorScriptPubKeyMan caches descriptor scripts and pubkeys in m_map_script_pub_keys and m_map_pubkeys to avoid regenerating them each time. It expands descriptors to generate new addresses using GetNewDestination(). Private keys are stored in m_map_keys unencrypted or encrypted in m_map_crypted_keys. It lazily generates FlatSigningProviders via GetSigningProvider() instead of on each call to avoid unnecessary work. Both classes can save wallet data to storage using functionality in WalletStorage.

Transaction Building

References: src/wallet/wallet.h

The CWallet class handles transaction building by constructing, signing, and broadcasting transactions from UTXOs owned by the wallet. CWallet stores all wallet transactions and UTXOs in a map indexed by transaction id (mapWallet) for quick lookups.

When a new transaction is requested, CWallet queries the m_chain interface to determine which UTXOs are owned and unspent. It then uses these UTXOs as inputs to fund the desired outputs for recipients. CWallet signs each input using the private keys associated with the addresses that received the UTXOs.

After signing, CWallet constructs a CMutableTransaction object containing all inputs, outputs, and signatures. This transaction is then broadcast to peers on the network using CWallet::CommitTransaction. Peers will validate the signatures and propagate the transaction.

CWallet tracks the transaction in mapWallet and notifies listeners that a new transaction was added. If the transaction is included in a block confirmed on the best chain, CWallet will update its UTXO set and balance. Unconfirmed transactions may be removed if replaced by a double spend on a longer chain.

Fee Estimation

References: src/wallet/feebumper.cpp, src/wallet/feebumper.h

The Bitcoin codebase implements fee estimation in several key ways. The feebumper namespace in …/feebumper.cpp contains the core logic for bumping transaction fees in the wallet. This allows replacing transactions with higher fee transactions to prioritize confirmation.

The PreconditionChecks function validates that a transaction can safely be replaced by ensuring it has no descendants, is not mined, and meets other criteria. The CheckFeeRate function checks that a provided fee rate meets minimum thresholds.

The critical CreateRateBumpTransaction function generates a new transaction with a higher fee for the specified original transaction. It handles creating matching inputs and outputs, calculates the new fee, and returns the result. EstimateFeeRate calculates an appropriate higher fee rate to use.

SignTransaction signs the generated replacement transaction using the wallet's private keys. CommitTransaction then submits this signed transaction to the network, updating wallet metadata to mark the original as replaced.

The SignatureWeights class tracks signature weights added during checking. Its AddSigWeight method accumulates these weights based on signature version. GetWeightDiffToMax returns the difference from the maximum weight. SignatureWeightChecker inherits from DeferringSignatureChecker and overrides CheckECDSASignature to call the base checker while updating the signature weights reference, allowing checking with weight tracking.

Wallet Storage

References: src/wallet/bdb.cpp, src/wallet/bdb.h, src/wallet/sqlite.cpp, src/wallet/sqlite.h

The Bitcoin wallet stores metadata like keys, addresses, transactions, and balances in databases for persistence across runs. The BerkeleyDatabase and SQLiteDatabase classes provide an interface for reading and writing this wallet data to the Berkeley DB and SQLite backends respectively.

The BerkeleyDatabase class represents a Berkeley DB database file within the environment. It inherits from the abstract WalletDatabase interface. Its constructor takes the BerkeleyEnvironment and file path. Methods like Open(), Close(), and Flush() delegate reads and writes to the underlying Berkeley DB handles. The MakeBatch() method returns a BerkeleyBatch object for performing transactions.

The BerkeleyBatch class manages transactions for reading and writing to the Berkeley DB. Methods delegate reads and writes via the underlying DB handles. It supports transactions via TxnBegin(), TxnCommit(), and TxnAbort(). This allows atomic operations on wallet metadata.

The SQLiteDatabase class represents an open SQLite wallet database file. It handles operations like opening, closing the database safely and providing a handle. The database uses SQLite's WAL journaling for crash safety. The SQLiteBatch class represents a batch of database operations that can be committed transactionally using prepared statements. This allows atomic operations on wallet metadata.

Wallet Encryption

References: src/wallet/crypter.cpp, src/wallet/crypter.h

The wallet encryption functionality in Bitcoin Core securely encrypts wallet contents and private keys using AES-256-CBC. This protects the contents of the wallet if the database files were to be accessed by an unauthorized party.

The CMasterKey class represents the master encryption key for the wallet. It contains the encrypted key, salt, and parameters for key derivation like the method (nDerivationMethod, usually BytesToKeySHA512AES()) and number of iterations (nDeriveIterations).

The CCrypter class handles the actual encryption and decryption of wallet data. It contains the encrypted encryption key (vchKey) and initialization vector (vchIV). The SetKeyFromPassphrase() method derives the encryption key from the passphrase using the BytesToKeySHA512AES() function, to strengthen the derived key.

The Encrypt() and Decrypt() methods on CCrypter perform AES-256-CBC encryption and decryption of arbitrary wallet data using the derived key and IV. There are also specialized functions like EncryptSecret() and DecryptSecret() for encrypting and decrypting private keys. When decrypting a private key, the public key is used as the IV to ensure the private key can only be decrypted by its intended recipient.

The wallet encryption functionality provides a secure way to encrypt private keys and wallet contents using AES-256-CBC. The use of password stretching like BytesToKeySHA512AES() and a per-key IV strengthens the security of encrypted private keys. This protects users' funds if an attacker were to access encrypted wallet files.

Wallet RPCs

References: src/wallet/rpc/wallet.cpp, src/wallet/rpc/wallet.h

The wallet RPC functionality is implemented primarily in the …/wallet.cpp file. This file defines RPCHelpMan objects which contain metadata for each RPC command, such as its name and parameters. It then implements each RPC by defining a lambda function that calls the appropriate methods on the CWallet class.

The CWallet class represents a Bitcoin wallet and contains methods for managing addresses, transactions, balances etc. Some key methods are getAddress, getBalance, sendCoins, and listTransactions. CWallet handles cryptographic signing and persisting data to the wallet database.

To retrieve the correct wallet instance for a request, the GetWalletForJSONRPCRequest() function is used. It first checks if the request refers to a loaded wallet by name. If not, it returns the wallet matching the node's configured wallet directory. This ensures each request operates on the correct wallet data.

The …/wallet.h file declares the CRPCCommand class which is used to define each RPC command. It exposes the list of commands via the GetWalletRPCCommands() function, which returns a span of CRPCCommand objects. Using a span instead of a standard container like vector improves performance.

The key classes implemented are CWallet for representing wallets, and CRPCCommand for defining each RPC. The GetWalletForJSONRPCRequest() function ensures requests use the right wallet. Metadata and business logic for each RPC is encapsulated via RPCHelpMan and lambda functions. This provides a centralized implementation of the wallet RPC interface.

Coin Selection

References: src/wallet/coinselection.cpp, src/wallet/coinselection.h

The …/coinselection.cpp file contains implementations of various algorithms for selecting UTXOs to fund transactions. A key class is OutputGroup, which represents a logical grouping of UTXOs like those from the same transaction. OutputGroup calculates values across the group efficiently.

Another important class is COutput, representing individual UTXOs. It stores metadata needed for selection. FilteredOutputGroups maps CoinEligibilityFilters to OutputGroupTypeMaps containing eligible OutputGroups.

Several algorithms are implemented to select optimal inputs given selection parameters. SelectCoinsBnB() uses a depth-first search tree to consider all combinations, prioritizing more valuable coins. SelectCoinsSRD() greedily selects coins randomly until the target is met. KnapsackSolver() uses a subset sum approximation algorithm.

These algorithms return a SelectionResult containing the selected UTXOs. SelectionResult stores the inputs and calculates values like total selected value and waste.

Utilities like GenerateChangeTarget() and ApproximateBestSubset() are used by the algorithms. GenerateChangeTarget() generates a random change amount. ApproximateBestSubset() approximates optimal subset sums.

Wallet Notifications

References: bitcoin

The Bitcoin Core software tracks changes to the wallet and notifies applications and the user about updates through the -alertnotify, -blocknotify, -walletnotify, and -shutdownnotify command line options. These options allow specifying scripts or executables to run when certain events occur.

The …/feature_notifications.py file contains tests that simulate real usage scenarios to validate the different notification options are working as expected. It defines the NotificationsTest class which inherits from the BitcoinTestFramework class. NotificationsTest sets up directories to receive notifications and parses the contents of files created in these directories to validate the expected data was provided.

Key events that should trigger notifications include:

The BitcoinTestFramework provides functionality for running multiple Bitcoin nodes to test interactions between nodes. Methods like set_test_params() allow passing extra command line arguments, while setup_network() handles node creation.

Functions such as expect_wallet_notify() validate the contents and data in notification output files match what is expected. This allows thorough testing of all the notification options and scenarios.

Wallet Tests

References: src/wallet/test

Unit and functional tests for wallets are located in the …/test directory. This directory contains comprehensive testing that validates wallet functionality in Bitcoin Core.

Tests are grouped into subdirectories based on what is being tested. The …/test directory contains unit tests that exercise low-level wallet components, such as …/walletload_tests.cpp which validates wallet descriptor loading. Files like …/wallet_tests.cpp focus on higher level wallet functionality and includes tests like scanning the blockchain for wallet transactions.

Fuzz testing is located in …/fuzz and is important for validating wallet code robustness. Files like …/coinselection.cpp expose the core coin selection algorithms like SelectCoinsBnB() and KnapsackSolver() to random inputs to find bugs. Balance accounting and notifications are fuzz tested by …/notifications.cpp which generates transactions between wallets over long block chains.

The WalletTestingSetup fixture in …/wallet_test_fixture.h is crucial for setting up the testing environment. It initializes a CWallet instance along with related components like a wallet loader and mock database. This allows tests to focus on wallet logic rather than environment setup.

Utilities in …/util.h and …/util.cpp further simplify testing. Functions like CreateSyncedWallet() initialize pre-configured wallets without rescanning.

Tests thoroughly exercise important classes like CWallet which manages wallet state, ScriptPubKeyMan for key and script management. Edge cases are explored around loading, rescanning, balance accounting and failure conditions to ensure robust functionality.

Cryptographic Primitives

References: src/crypto, src/secp256k1

The Bitcoin codebase implements a variety of cryptographic primitives needed for securing transactions and validating the blockchain. This includes hash functions, digital signatures, public-key cryptography, and encryption algorithms.

The …/crypto directory contains implementations of cryptographic algorithms and data structures used throughout Bitcoin. It provides hash functions like SHA-256 and RIPEMD-160 for integrity checks. Symmetric ciphers like AES-256 in CBC mode allow encrypting arbitrary lengths of data. The HMAC-SHA256 construction provides message authentication. The Poly1305 algorithm enables message authentication codes.

The …/secp256k1 directory contains a fast elliptic curve cryptography library implementing the secp256k1 curve. It provides constant-time ECDSA signing and verification for the Bitcoin protocol. The secp256k1_context object manages cryptographic operations. Elliptic curve points are represented by the secp256k1_ge class in affine coordinates and secp256k1_gej in Jacobian projective coordinates. Scalar multiplication uses a precomputed table for windowed NAF representation. Field arithmetic like modular addition and multiplication is implemented efficiently via secp256k1_fe.

The …/ctaes directory contains a constant-time implementation of AES. It represents the AES state as AES_state containing uint16_t slices. The SubBytes() function implements the S-box substitution circuit-based. MixColumns() performs column mixing via polynomial multiplication on the slices.

The …/chacha20.cpp file implements the ChaCha20 stream cipher. The ChaCha20 class handles encryption of arbitrary lengths by caching keystream blocks. FSChaCha20 provides forward secrecy by periodically rekeying the underlying ChaCha20 instance.

The …/poly1305.cpp file implements the Poly1305 MAC algorithm. The poly1305_context struct stores the authentication state like r, h, pad fields. poly1305_init() initializes this from a key. poly1305_update() processes message blocks. poly1305_finish() outputs the tag.

Hash Functions

References: src/crypto/sha256.cpp, src/crypto/sha256.h, src/crypto/ripemd160.cpp, src/crypto/ripemd160.h

The Bitcoin codebase implements several cryptographic hash functions that are used throughout the system. The …/crypto directory contains optimized reference implementations of SHA-256 and RIPEMD-160.

The …/sha256.cpp file provides a C++ class called CSHA256 for hashing using SHA-256. CSHA256 allows incremental hashing of data by writing input with Write() and finalizing the hash with Finalize(). It also supports resetting the context with Reset(). This class is used heavily in Bitcoin for tasks like block and transaction hashing.

The file also contains a function called SHA256D64() that implements an optimized version of double SHA-256 hashing required for block headers. It processes multiple 64-byte blocks in parallel using SIMD instructions if available.

Similarly, …/ripemd160.cpp defines a CRIPEMD160 class for RIPEMD-160 hashing. It follows the RIPEMD-160 specification, hashing input data in 64-byte blocks with the Transform() function. CRIPEMD160 exposes Write(), Finalize(), and Reset() methods to match the SHA-256 interface.

Both implementations leverage auto-detection of CPU features like SSE4, AVX2, and ARM SHA extensions to select optimized implementations of the underlying compression functions for improved performance.

Elliptic Curve Cryptography

References: src/secp256k1/include, src/secp256k1/src

The …/secp256k1 library implements cryptographic functionality needed for the Bitcoin protocol, including elliptic curve cryptography algorithms like ECDSA signing and verification, ECDH key exchange, Schnorr signatures, and Elligator-Swift encoding. It provides optimized and constant-time implementations of elliptic curve arithmetic, scalar multiplication, and cryptographic primitives over the secp256k1 curve.

The core elliptic curve functionality is implemented in files under …/group_impl.h and …/ecmult_impl.h. These files define classes like secp256k1_ge and secp256k1_gej to represent points on the curve, and implement algorithms for performing group operations between points. Functions like secp256k1_gej_add_var and secp256k1_gej_double_var efficiently add and double points using Jacobian projective coordinates. secp256k1_ecmult_const implements constant-time scalar multiplication by precomputing multiples of the generator point and using a windowed NAF representation of the scalar.

The …/scalar_impl.h file implements scalar arithmetic needed for operations like ECDSA signing. It represents scalars as 256-bit integers split across limbs, and provides functions for basic operations like addition, multiplication, and reducing scalars modulo the group order.

ECDSA signing and verification functions are implemented in …/ecdsa_impl.h. This includes functions like secp256k1_ecdsa_sign to generate signatures and secp256k1_ecdsa_verify to validate signatures. It supports signatures in a recoverable format via secp256k1_ecdsa_recoverable_signature which allows public key recovery.

The …/main_impl.h file implements Schnorr signatures as defined in BIP-340, providing signing and verification functions like secp256k1_schnorrsig_sign32 and secp256k1_schnorrsig_verify. It uses a deterministic nonce generation function.

The …/main_impl.h file contains an implementation of Elliptic Curve Diffie-Hellman key exchange via the secp256k1_ecdh function, which performs a constant-time ECDH computation.

Encryption and Authentication

References: src/crypto/aes.cpp, src/crypto/aes.h, src/crypto/hmac_sha256.cpp, src/crypto/hmac_sha256.h

This section covers symmetric encryption, message authentication codes (MACs), and authenticated encryption. Symmetric encryption uses secret keys to encrypt and decrypt data confidentially. MACs allow verification of data integrity using cryptographic hash functions. Authenticated encryption combines these to provide both confidentiality and integrity.

The …/aes.cpp and …/aes.h files implement AES-256 symmetric encryption. The AES256Encrypt and AES256Decrypt classes handle encryption and decryption of 16-byte blocks under an AES key. For streams, the AES256CBCEncrypt and AES256CBCDecrypt classes perform cipher block chaining (CBC) mode. CBC XORs each plaintext block with the previous ciphertext block before encryption to prevent patterns.

The …/hmac_sha256.cpp and …/hmac_sha256.h files implement HMAC using SHA-256 for message authentication. The CHMAC_SHA256 class performs the HMAC construction. It contains inner and outer CSHA256 hash objects. The constructor derives the HMAC keys from the input key. Write() hashes the data with the inner key, while Finalize() hashes the inner hash with the outer key to produce the HMAC.

These files provide the building blocks for symmetric encryption, hashing MACs, and combining them for authenticated encryption of Bitcoin protocol data and wallet contents. The constant-time AES and HMAC implementations help prevent side-channel attacks.

Constant Time Implementations

References: src/crypto/ctaes, src/secp256k1/src/scalar_impl.h

Constant Time Implementations focuses on preventing timing attacks and side channels. The core functionality implemented includes:

The …/ctaes directory contains a constant-time AES implementation written in C. It represents the AES state as uint16_t slices in an AES_state struct, allowing AES operations to be performed via bitwise logic on these slices. Key functions include AES128_init() to initialize encryption contexts from keys, and AES128_encrypt()/AES128_decrypt() to encrypt or decrypt blocks.

The SubBytes() function implements the AES S-box substitution via a circuit-based approach using Boolean logic operations. This is done in a constant-time manner without data-dependent branches. MixColumns() also implements column mixing for AES in constant-time by rotating slice bits and computing values via polynomial multiplication modulo x^4 + 1.

The secp256k1 scalar arithmetic in …/scalar_impl.h also aims to prevent side channels. The secp256k1_scalar_set_b32_seckey() function sets a scalar from a secret key in constant-time. secp256k1_scalar_split_lambda() splits a scalar for point multiplication while bounding results, important for constant-time implementations.

These implementations aim to prevent timing attacks and other side channels by carefully designing algorithms and low-level logic to avoid data-dependent branches or memory accesses. This allows cryptographic operations to complete in a constant elapsed time regardless of secret data values.

Key Derivation and Generation

References: src/crypto/hkdf_sha256_32.cpp, src/crypto/hkdf_sha256_32.h

The …/hkdf_sha256_32.cpp and …/hkdf_sha256_32.h files implement HMAC-based Key Derivation Function (HKDF) using HMAC-SHA256 as specified in RFC 5869. The CHKDF_HMAC_SHA256_L32 class handles the full HKDF key derivation process.

The class constructor CHKDF_HMAC_SHA256_L32() takes an input keying material (IKM) and salt as arguments. It derives the pseudorandom key (PRK) by running the IKM and salt through CHMAC_SHA256.

The Expand32() method implements the HKDF Expand step. It takes the previously derived PRK, an info parameter, and expands the PRK into 32 bytes of output keying material (OKM) using CHMAC_SHA256 with a counter.

No error handling is implemented and it is assumed inputs are valid. The PRK and OKM sizes match Bitcoin's 32 byte secret size for compatibility. The class provides a simple interface for HKDF in Bitcoin without a general implementation.

The files also reference deterministic nonce generation as specified in RFC 6979. However, the summaries do not provide details on this, so it is not covered further here. Details may be found in the Cryptographic Primitives section.

Utility Functions

References: src/crypto/common.h, src/secp256k1/src/util.h

The …/common.h file contains utility functions for common tasks like reading and writing variable-length integers in little-endian and big-endian format to and from byte arrays using functions like ReadLE16(), WriteBE64(), and others. It also contains the CountBits() function for efficiently counting the number of bits set in an integer value.

The …/util.h file provides foundational portable utilities for elliptic curve cryptography. It defines the secp256k1_callback struct used to pass callback functions and data for error and argument handling. It contains integer support macros that select between 128, 64, or 32-bit integer implementations depending on compiler and platform support. For platforms where compiler builtins are unavailable, it implements the variable-time secp256k1_ctz32_var population count algorithm as a fallback.

Both of these files contain small inline utility functions and data structures that provide an important low-level foundation for common tasks in the Bitcoin codebase. Functions like ReadLE16() and CountBits() eliminate the need to reimplement basic byte manipulation and integer algorithms throughout the code. Similarly, abstractions like secp256k1_callback allow higher-level ECC code to remain portable across platforms. These utilities focus on common crypto primitives, data encodings, and other low-level tasks that likely see widespread reuse.

Tests and Benchmarks

References: bitcoin

The …/bench directory contains benchmarks for testing and measuring the performance of cryptographic implementations used throughout Bitcoin Core. The benchmarks help validate cryptographic functionality meets performance requirements across platforms and identify any bottlenecks.

The benchmarks are implemented by subclassing the benchmark class defined in …/bench.h. The BENCHMARK macro registers benchmarks with the benchmarking framework. Command line arguments in benchmark::Args allow controlling benchmark execution. The core benchmarking framework is defined in …/bench.cpp, which runs all registered benchmarks based on specified criteria. It handles benchmark filtering, output, and statistics collection.

Benchmarks are organized into subdirectories based on the cryptographic primitive tested. Functions from libraries like …/crypto and …/secp256k1 that implement cryptographic operations are benchmarked to measure performance, such as secp256k1_ecdsa_verify() and secp256k1_ec_pubkey_parse().

Benchmarks generate randomized test vectors without vulnerabilities from crafted inputs. Precise control and batching helps average results over many iterations for accurate performance data. Maintainers use these benchmarks to prioritize efficiency improvements and ensure cryptographic performance meets requirements across platforms.

Configuration and Initialization

References: src/init, src/node, src/interfaces

The core functionality for configuration and initialization in Bitcoin Core is handled through the interfaces::Init interface and initialization classes that implement it. The interfaces::Init interface defines common methods that initialization classes must implement, such as makeNode(), makeChain(), and makeWalletLoader(). This allows initialization logic to be decoupled from different applications and implemented independently.

The key initialization classes include BitcoinNodeInit, BitcoinGuiInit, and BitcoinQtInit. BitcoinNodeInit is defined in …/bitcoin-node.cpp and is responsible for initializing core node components. Its constructor takes a NodeContext reference, which provides configuration passed throughout initialization. Methods like makeNode(), makeChain(), etc call interface factory functions while passing the context.

BitcoinGuiInit defined in …/bitcoin-gui.cpp initializes interfaces for the GUI like Node, Chain, WalletLoader, and Echo. Its constructor initializes a NodeContext and stores references to interfaces. Factory methods pass the context to initialized interface implementations.

BitcoinQtInit defined in …/bitcoin-qt.cpp handles GUI application initialization by implementing Init. Its constructor initializes a NodeContext and associates itself with the node.

The NodeContext class defined in …/context.h and …/context.cpp manages objects and services needed beyond the P2P network and validation, including the address manager, ban manager, chain interface, kernel context, network components, mempool, and more. It provides references to these objects throughout initialization without using global variables.

Common initialization functionality is defined in …/common.cpp and …/common.h. This includes functions for adding and setting command line arguments related to logging via the ArgsManager class. The ArgsManager manages command line arguments but its implementation is abstracted.

The MakeNodeInit() and MakeGuiInit() functions return initialized Init instances to allow retrieving other initialized interfaces. This ties the different interfaces and components together for initialization.

Initialization

References: src/init/bitcoin-node.cpp, src/init/bitcoin-wallet.cpp, src/init/bitcoin-gui.cpp, src/init/bitcoin-qt.cpp

The core initialization logic is handled by three main classes: BitcoinNodeInit, BitcoinGuiInit, and BitcoinQtInit. BitcoinNodeInit is defined in …/bitcoin-node.cpp and initializes the Node interface, Chain interface, WalletLoader interface, and Echo interface. Its constructor takes a NodeContext reference which is used to pass configuration and chain state to initialized interfaces. The makeNode() method calls the interfaces::MakeNode function to retrieve an initialized Node implementation.

BitcoinGuiInit is defined in …/bitcoin-gui.cpp and initializes interfaces for the GUI via makeNode(), makeChain(), etc. Its constructor initializes the NodeContext which is used to pass state to interfaces.

BitcoinQtInit handles Qt application initialization and is defined in …/bitcoin-qt.cpp. Like the others, it inherits from interfaces::Init and overrides methods to initialize the required interfaces via their factory functions. Its constructor initializes the NodeContext and associates the initializer.

Each class provides access to initialized interfaces via methods like makeNode(). This allows other code to retrieve fully configured implementations of interfaces like Node and Chain through the initialization context object. The context encapsulates common configuration that the interfaces need.

Argument Parsing

References: bitcoin

The Bitcoin Core codebase handles parsing command line arguments through the ArgsManager class defined in …/args.cpp and …/args.h. The ArgsManager class centralizes argument parsing. When Bitcoin Core starts up, it initializes an ArgsManager instance. The ArgsManager allows arguments to be registered by calling AddArg() and specifying the argument name, help text, and type. This registration system defines the valid arguments. Later, ParseParameters() is called to parse the actual command line and configuration file arguments. It uses the registration rules to parse arguments into a Settings object based on argument name and type.

The Settings object stored within ArgsManager aggregates the parsed arguments and settings from multiple sources like the command line and configuration file. It handles precedence between these sources, with command line arguments taking highest precedence. The Settings object also supports negation of boolean values set on the command line to override configuration settings.

Logging

References: src/init/common.cpp, src/init/common.h

The …/common.cpp and …/common.h files handle configuring the logging output for Bitcoin applications. These files provide a centralized way to control logging behavior across node, wallet, and other software by parsing command line arguments and initializing the global logging subsystem.

The AddLoggingArgs() function in common.cpp allows configuring logging options via command line flags like the log file location, categories, and log levels. These arguments are added to an ArgsManager object which parses the command line input.

The SetLoggingOptions() function then sets properties on the global LogInstance object based on the values parsed from the command line. This configures aspects like whether logging output goes to file or console, whether timestamps are included, and other runtime options.

Log levels can be set globally or for specific categories using the SetLoggingLevel() function. It parses the -loglevel flag to determine the desired logging threshold.

Individual logging categories can be enabled or disabled through the -debug and -debugexclude flags, which are processed by SetLoggingCategories().

Finally, StartLogging() initializes the actual logging output. It opens the log file if specified, prints startup messages, logs the config file path, and begins logging messages based on the configured levels and categories.

The LogInstance represents the global logging subsystem. Methods are called on it to control the logging destination, levels, timestamps, and included categories. This provides a centralized interface for configuring logging across the different Bitcoin software components.

Configuration

References: bitcoin

The Bitcoin Core software can be configured through command line arguments and configuration files. Key configuration functionality is provided by the ArgsManager class defined in …/args.cpp and …/args.h.

The ArgsManager class centralizes argument parsing. Arguments are registered with AddArg() which specifies help text and type. ParseParameters() then parses arguments into the Settings object based on these rules. This provides a clean interface via accessors like GetArg() while handling complex tasks like precedence and negation.

Configuration values can be retrieved by calling GetSetting() which merges sources like command line arguments, configuration files, and default values in a specific precedence order. This allows flexibility in how nodes are configured.

Shutdown

References: bitcoin

The Bitcoin Core codebase handles gracefully shutting down processes. When a shutdown signal is received, the shutdown logic is called. This logic handles cleanup tasks in the reverse order of initialization. First it stops any running threads and nodes. Then it flushes the wallet if present. Next it flushes the block index to disk to persist any changes. The UTXO database is instructed to flush to disk. Finally logging and debug output is stopped and the shutdown process is completed in other modules.

By scheduling the shutdown logic after any in-progress work, it allows that work to complete before stopping threads and flushing data. The reverse initialization order ensures resources are cleaned up before dependent code or data is finalized. This graceful shutdown process helps ensure the node state on disk remains consistent even if an unexpected restart occurs due to system shutdown or crash.

Utility Libraries

References: src/support, src/script, src/primitives

The …/support directory contains important utility code that provides common low-level functionality used throughout the Bitcoin codebase. This includes crucial components for memory management, events handling, and other algorithms and data structures.

The …/allocators subdirectory implements custom memory allocators for securely managing sensitive memory in Bitcoin. The PoolResource class in …/pool.h manages memory pools with freelists of different block sizes, improving allocation efficiency. The secure_allocator in …/secure.h locks allocated pages and securely erases memory before freeing.

The …/events.h file provides RAII wrappers for libevent objects like event_base and evhttp that ensure thread-safe cleanup. Functions like obtain_event_base() simplify constructing and managing these objects.

The …/lockedpool.cpp and …/lockedpool.h files implement the LockedPool class for managing locked memory allocation. LockedPool uses Arena objects to manage allocated memory regions with a best-fit strategy. The Arena class manages contiguous locked memory divided into chunks using maps to track free chunks. LockedPageArena subclasses handle platform-specific memory locking from a LockedPageAllocator. LockedPool starts with one LockedPageArena but can grow more as needed.

The …/cleanse.cpp and …/cleanse.h files contain the memory_cleanse() function, which securely wipes memory before freeing in a way that cannot be optimized out. This helps protect sensitive application secrets and keys.

Memory Management

References: src/support/allocators, src/support/lockedpool.cpp, src/support/lockedpool.h

The Bitcoin codebase provides custom memory allocators and utilities for securely managing memory. This is implemented in the …/allocators directory.

The …/pool.h file defines the PoolResource class, which manages multiple memory pools of different block sizes using freelists. It allocates large chunks of memory from the system and carves out blocks to service allocation requests, adding any leftover memory to the freelists. The PoolAllocator class forwards allocation and deallocation requests to a PoolResource instance, allowing containers to use the memory pools.

The …/secure.h file provides a secure_allocator that locks allocated pages to prevent paging out. It securely clears memory using memory_cleanse() before freeing. This secure_allocator is used by SecureString and secure_unique_ptr to securely manage string and object memory.

The …/zeroafterfree.h file defines a zero_after_free_allocator that clears memory after deallocation using memory_cleanse(). The SerializeData type uses this allocator to securely erase bytes upon destruction.

The …/lockedpool.cpp file implements the LockedPool class, which manages memory allocation and locking. It uses Arena objects employing best-fit allocation to manage memory regions. The Arena class maintains maps and multimaps to track used and free memory chunks. The LockedPoolManager singleton provides global access to the locked memory pool.

Utilities

References: src/support/events.h, src/support/cleanse.cpp, src/support/cleanse.h

This section covers common utilities used throughout the Bitcoin codebase. Two important utilities are the events library and memory cleanse functions.

The …/events.h file provides wrappers and utilities for working with the libevent library in a thread-safe manner. It defines RAII wrappers for common libevent objects like event_base and evhttp that automatically free the underlying objects on destruction. Functions like obtain_event_base() and obtain_evhttp() construct the libevent objects and return them wrapped in these smart pointers. This simplifies object lifetime management and avoids memory leaks.

The memory_cleanse() functions in …/cleanse.cpp and …/cleanse.h provide a way to securely wipe sensitive data from memory. The memory_cleanse() function takes a pointer and length to a memory region and overwrites the data with zeros. It uses platform-specific techniques like SecureZeroMemory() on Windows and std::memset() with compiler barriers on other platforms to ensure the memory is really wiped and not optimized away. This function helps address security and privacy concerns by preventing secret keys or passwords from remaining in memory after use.

Script Evaluation

References: src/script/interpreter.h, src/script/interpreter.cpp, src/script/script.h, src/script/script.cpp

Script evaluation is handled through the CScript and CScriptWitness classes defined in …/script.h. The core logic that evaluates scripts is the EvalScript function defined in …/interpreter.cpp. It takes a stack and script as input, and iterates through each opcode, executing logic like pushing values, conditionals, etc. This modifies the stack.

Signature validation opcodes invoke functions to handle signature checking. The GenericTransactionSignatureChecker class defined in …/interpreter.h is used to do the actual signature checking. It supports checking both ECDSA and Schnorr signatures. These functions extract the necessary transaction data, hash it, and validate the signature.

For Taproot, the SignatureHashSchnorr function handles correctly hashing the transaction data for Schnorr signature validation. This optimizes caching of prevout/sequence hashes.

Witness program scripts are executed via the ExecuteWitnessScript function. This runs the script while enforcing clean stack rules and size limits appropriate for witness execution.

Cryptographic Primitives

References: src/crypto, src/secp256k1

The secp256k1 library provides implementations of cryptographic primitives needed for elliptic curve cryptography on the secp256k1 curve. It includes efficient implementations of ECDSA signing and verification, as well as scalar multiplication for secret key operations.

ECDSA signatures are generated and verified using the secp256k1_ecdsa_sign and secp256k1_ecdsa_verify functions defined in …/secp256k1.h. These functions operate on an initialized secp256k1_context object, which manages cryptographic state and callbacks.

Scalar multiplication is used to derive the public key from a secret key. It is implemented using a windowed Non-Adjacent Form (NAF) representation of the scalar and precomputed tables for multiples of the generator point G. This allows general multiplication to be performed quickly via functions like secp256k1_ec_pubkey_create defined in secp256k1.h.

Elliptic curve points are represented using secp256k1_ge for affine coordinates and secp256k1_gej for Jacobian projective coordinates, defined in …/secp256k1.h. Constant-time implementations of functions like secp256k1_gej_add_var in …/group_impl.h perform addition and doubling of curve points.

Field elements secp256k1_fe defined in …/field.h represent numbers modulo the secp256k1 curve field. Field arithmetic operations like addition and multiplication are implemented in constant-time functions such as secp256k1_fe_add and secp256k1_fe_mul defined in …/field_impl.h.

Serialization

References: src/primitives/transaction.h, src/primitives/transaction.cpp, src/primitives/block.h, src/primitives/block.cpp

The core Bitcoin blockchain primitives like transactions, blocks, and their components need to be serialized and deserialized in order to be transmitted over the network or stored persistently. The …/primitives directory contains the key code for serializing these primitives in a standardized format.

The CTransaction, CTxIn, CTxOut, and CMutableTransaction classes in …/transaction.h define the data structures used to represent transactions. CTransaction represents a complete transaction, while CTxIn and CTxOut define the input and output components. CMutableTransaction is used for building transactions before finalizing to CTransaction.

These classes leverage serialization functionality to encode and decode transactions to streams in a binary format. For example, CTransaction has a Serialize method that serializes the transaction according to the consensus rules. Another method deserializes a parsed transaction back into a CTransaction object by reading elements from the serialized byte stream.

The CBlock, CBlockHeader classes in …/block.h provide similar serialization functionality for blocks and headers. The GetHash() method on CBlockHeader computes the double-SHA256 hash of the serialized block header.

Overall, the primitives classes define the data model while common serialization logic allows transactions and blocks to be encoded and transmitted over the network or to disk, in a standardized format defined by the Bitcoin protocol.

Build System and Continuous Integration

References: ci, contrib/guix

The Bitcoin Core codebase uses a robust continuous integration (CI) system to build, test, and verify changes in a reproducible manner. This is implemented through the ci directory, which contains scripts and configuration for running CI tasks on commits and pull requests.

Some of the key aspects of the CI system include:

  • Leveraging Docker containers and dependency tools like depends for isolated, reproducible environments between developers and across versions. This ensures consistent builds over time.

  • Modularization of scripts into stages like test_run_all.sh and subdirectories for each task (e.g. …/lint, …/test). Individual scripts handle discrete steps.

  • Environment variables configure platforms, dependencies, caches, and other options. For example, HOSTS controls operating systems built and JOBS sets parallel build jobs.

  • Utilities like retry in …/retry abstract out retrying commands for robustness against transient errors.

  • The Git commit range is determined based on environment variables and used across scripts for consistency.

  • Scripts pin Guix versions using time-machine from …/prelude.bash to ensure reproducible builds over time.

  • Platform-specific setup scripts like …/00_setup_env_native_asan.sh configure packages, flags, and build options for running tests on that platform.

  • Binary signing is implemented in …/codesign.sh using detached signatures from Git. Signatures are applied and packaged for different platforms.

  • Final binaries, packages, and artifacts are moved to the …/build.sh output directory once built and tested.

Build Scripts

References: ci/lint_run_all.sh, ci/test_run_all.sh, ci/lint/04_install.sh, ci/test/00_setup_env_native_asan.sh

The Bitcoin Core build scripts handle installing dependencies, running tests, and compiling the code as part of the continuous integration (CI) process. Scripts like …/lint_run_all.sh and …/test_run_all.sh orchestrate the overall linting and testing workflows.

…/lint_run_all.sh sets environment variables and sources scripts to install linting dependencies and run the actual linting tests. It exports LC_ALL for UTF-8 locale, sets errexit to exit on failures, and sources 00_setup_env.sh and 04_install.sh. 04_install.sh installs common dependencies like curl and xz-utils using apt-get. It also installs Python using pyenv along with build dependencies. The Rust-based lint test runner is compiled from source in …/lint. Python packages for linting like flake8 and mypy are installed using pip. ShellCheck is downloaded for shell script analysis.

…/test_run_all.sh sets LC_ALL and sources 00_setup_env.sh to initialize environment variables. It runs 02_run_container.sh to execute tests inside a Docker container. 00_setup_env.sh sets variables and paths while 02_run_container.sh builds and runs the Docker container containing the test suite.

…/00_setup_env_native_asan.sh prepares the environment for running tests with AddressSanitizer. It sets variables like CONTAINER_NAME and PACKAGES, and installs packages such as clang and llvm via apt-get. It also sets compiler flags for features like C++20 and sanitizers in the BITCOIN_CONFIG variable.

Testing Infrastructure

References: ci/test/00_setup_env_native_asan.sh, ci/test/02_run_container.sh, ci/test/03_test_script.sh

The Bitcoin test infrastructure is implemented in several key files and directories within the repo. Unit tests are contained in …/test and test low-level components in isolation. Functional tests that exercise full node and wallet functionality end-to-end are in …/functional.

The continuous integration system for running automated tests is defined in ci. This directory contains scripts for setting up build environments, running tests, and processing results. Specifically, …/00_setup_env_native_asan.sh prepares a Ubuntu environment with the necessary compiler, libraries, and flags to build Bitcoin Core with various sanitizers enabled. This allows memory errors and bugs to be detected during testing.

…/02_run_container.sh handles executing the tests within a Docker container in a robust and reproducible way. It exports environment variables, builds the test image, mounts volumes, and runs initialization, dependency, and test scripts. Finally, …/03_test_script.sh defines the main test script that is executed. It sets up sanitizers, prints system info, determines the host system, patches LevelDB, installs dependencies, builds Bitcoin, and runs the unit and functional test suites. Optional sections also execute fuzz, linting, and include tools.

Linting

References: ci/lint/04_install.sh, ci/lint/06_script.sh

The linting process ensures code quality and catches bugs or issues early by running static analysis tools and code linters over the source code. Linting is performed on every code change by the continuous integration system.

The main linting and analysis tasks are configured and run from scripts in the …/lint directory. The …/04_install.sh script handles installing dependencies and tools like Python and pip. It also compiles the Rust-based lint test runner from source at …/test_runner.

This test runner is the primary tool for running linters. It is designed to run linters in parallel processes for performance. The linting logic for tools is implemented in Rust.

The …/06_script.sh script executes the linting process. It sets variables and runs the test_runner binary, passing modified files. This allows linting only changed code. It also runs commit checks.

In summary, the linting process installs dependencies, compiles the test runner, and executes linting tools in parallel over changed code on every commit. This catches issues early and ensures code quality.

Reproducible Builds

References: contrib/guix, contrib/guix/libexec/build.sh

The Bitcoin project achieves reproducible builds across machines using the Guix functional package manager. Guix handles all dependencies deterministically, building packages from source in a clean, reproducible manner.

The …/guix directory contains scripts and utilities that integrate deeply with Guix to build Bitcoin Core and its dependencies. The main …/build.sh script performs the actual build using Guix. It sets up a pinned Guix environment with the time-machine function to ensure consistent builds over time. Environment variables like HOSTS and JOBS allow customizing the build process.

The build script first builds compiled dependencies like OpenSSL and Boost in the "depends" tree. It then configures and builds Bitcoin Core itself for the target platform specified in $HOST, installing to a staging directory. Binaries are then deterministically packaged into final release artifacts like zip packages and installers. Validation checks are performed on the built binaries.

The …/prelude.bash file sets up common variables and functions used by the build scripts. It validates prerequisites, pins the Guix version, and defines reusable variables for versioning, paths and other build metadata. This ensures clean separation of versions and locations for reproducible builds.

In summary, the Guix integration achieves fully reproducible Bitcoin Core builds across machines by handling all dependencies, building from a pinned environment, and packaging outputs deterministically. Environment variables provide build customization.

Utilities

References: ci/retry

The …/retry directory contains utilities for retrying shell commands multiple times with increasing delays between attempts. This helps avoid overloading the system being retried in cases where failures may be transient.

The core functionality is provided by the retry shell script. retry allows running any command a configurable number of times with either exponential backoff or constant delays between retries. It tracks the number of attempts using an internal counter variable. On each attempt, it runs the command and checks the exit code. If non-zero, it calls the backoff() function to calculate the delay time using the counter and delay parameters. It then sleeps for that duration before retrying.

The backoff() function implements an exponential backoff algorithm. It calculates increasing delays between attempts using the RETRY_ATTEMPT counter and minimum, maximum, and base delay values passed to it. This avoids overloading the system with rapid retries.

retry returns the exit code of the last run command. It also allows running a custom "fail script" using the -f option if all retry attempts fail. This script has access to the RETRY_ATTEMPT variable containing the last attempt number.

The key algorithms and components are implemented in the retry() and backoff() functions without using classes. retry() handles the main logic, while backoff() focuses only on calculating increasing delays. This separation of concerns keeps the code well organized and easy to understand.

Documentation

References: doc

The Bitcoin Core documentation contains important information for both users and developers. For developers, the documentation includes developer notes, release notes, and specifications that provide technical details on the implementation and evolution of Bitcoin Core.

Developer notes give insight into the internal architecture and design of Bitcoin Core. For example, notes on the CCheckQueue class explain how it enables parallel script validation of transactions in blocks through its Loop() method. Release notes document the changes between each version of Bitcoin Core. For example, release notes for version 0.9.0 describe how the wallet functionality was improved through updates to the walletpassphrase RPC command behavior. Specifications define protocols and standards that are implemented in Bitcoin Core. For example, BIPs that are implemented specify the low-level network protocol handled by functions in src/protocol.h.

The documentation in doc contains several important files:

  • …/developer-notes.md provides an overview of key classes like CConnman for network handling and CWallet for wallet operations.

  • …/release-notes includes release notes files for each version that summarize new features, fixes, and changes at a high level.

  • Individual files for specifications implemented in Bitcoin Core, such as …/bips.md which documents support for BIPs and the relevant code changes.

The documentation aims to help both users and developers navigate the capabilities and evolution of the Bitcoin Core software over time. For developers, it provides insight into the internal design and assists with tasks like upgrading to new versions safely.

Chain State

References: src/chain.h, src/chain.cpp

The CChain class manages the main blockchain and represents its current state. It stores pointers to the CBlockIndex of the tip block, allowing efficient traversal of the chain to find ancestors and locate forks. During initial block download, CChain is used to extend the local chain by connecting new blocks. The SetTip() method updates the tip pointer as blocks are processed.

The CBlockIndex class represents each block in the chain, storing its metadata like height, hash, and pointers to its parent and skip blocks. The pprev and pskip pointers allow fast traversal of the chain to find ancestors or locate forks. Each CBlockIndex represents a node in the chain, forming a linked list structure as blocks are connected.

The UTXO set and other chain state is maintained through processing blocks with CChain. As each block's transactions are validated, the effects are applied to the UTXO set and other chain-level metadata by calling methods like DisconnectBlock() and ConnectBlock(). This updates the set of unspent transaction outputs and validates the blockchain rules have not been broken. During initial block download, this process populates the local chain state from the beginning of the blockchain. In normal operation, it updates the state in response to incoming blocks and reorganizations.

The GetLocator() method on CChain generates a block locator for catching up. It returns a compact list of block hashes by traversing the chain exponentially towards the genesis block. Peers can use this to efficiently determine common ancestors and locate potentially missing blocks. The FindFork() method finds the last common block between a block and the main chain, allowing reorganizations to be processed correctly.

P2P Protocol

References: src/protocol.h

The CMessageHeader class represents the header attached to all P2P messages in Bitcoin. It contains fields for the message start characters, command string, size, and checksum. The constructor validates the command string length. Serialization methods encode and decode the header contents.

The NetMsgType namespace defines string constants for all message types like VERSION and INV. ServiceFlags define node capabilities as an enum class. These are advertised in messages to describe supported features.

The CAddress class extends CService with additional metadata like services and last seen time. It supports both V1 and V2 serialization formats, conditionally serializing service flags as 64-bit integers or compact sizes. The disk format also encodes the serialization version.

The CInv class represents an inventory vector, used to advertise or request objects. The type is one of the GetDataMsg types. Serialization and comparison operators are provided.

The low-level P2P protocol handles block and transaction relay between nodes. The CMessageHeader allows encoding and decoding P2P messages. NetMsgType defines standard message types like requesting inventory data with GETDATA. CAddress objects carry peer metadata and CInv represents requested inventory vectors. Nodes advertise their capabilities with ServiceFlags.

RPC Interface

References: src/rpc/server.h, src/rpc/server.cpp

The CRPCTable class in …/server.h and …/server.cpp handles routing incoming JSON-RPC requests to the appropriate handler functions. It maintains a map of command names to CRPCCommand objects, where each CRPCCommand represents an individual RPC method and contains metadata like the name, category, arguments, and handler function. The CRPCTable's execute() method looks up the requested command name and calls its handler.

Some key handlers include getrpcinfo which returns metadata about the RPC server, and stop which signals the RPC server to shut down gracefully. The help() handler generates help text by iterating through the command map. CRPCTable also provides utilities like listCommands().

The RPCTimerInterface and deadlineTimers in …/server.h manage delayed RPC executions via timers. The RPCRunLater() function in …/server.cpp schedules the timer and handler. This allows asynchronous or periodic tasks to be run through the RPC layer.

The CRPCCommand struct represents an individual RPC method and contains the name, category, list of arguments, and std::function handler that gets called to process each request. Multiple handlers can be supported for a single command by using the handler function as a dispatcher.

Wallet Encryption

References: src/wallet/crypter.cpp, src/wallet/crypter.h

The …/crypter.cpp and …/crypter.h files contain code for encrypting wallet contents and securing private keys. This is done through the CCrypter class, which handles encryption and decryption of wallet data.

The CCrypter class manages the encryption key and initialization vector, stored as member variables vchKey and vchIV. It provides Encrypt() and Decrypt() methods to encrypt and decrypt arbitrary data passed as CKeyingMaterial objects. The encryption key and IV are derived from the user's passphrase using the BytesToKeySHA512AES() function.

The CMasterKey class represents the encrypted master encryption key for the wallet. It contains the encrypted key, salt, and parameters for the key derivation method, usually nDerivationMethod, and number of iterations nDeriveIterations.

Specialized functions like EncryptSecret() and DecryptSecret() handle encrypting and decrypting private keys. When decrypting a private key, the public key is used as the IV to ensure the private key can only be decrypted by its intended recipient.

The CCrypter provides a secure way to encrypt wallet contents like private keys using AES-256 in CBC mode. Password stretching helps protect against brute force attacks by slowing down attempts to guess the passphrase. Storing encryption parameters in the CMasterKey allows decrypting wallet files across sessions. Together these provide a robust system for encrypting Bitcoin wallet contents and keys.

Blockchain Reorganization

References: src/txmempool.cpp

When the node receives a new block, it must validate the block and update its local blockchain state if the block is valid. However, sometimes a block is received that builds on an invalid part of the chain, creating a temporary fork. In this case, the node may receive another block building on a different part of the chain, causing a reorganization as the longer chain becomes the new best chain.

During a reorganization, the node uses src/validation.cpp to handle rolling back and replaying transactions. When a reorganization occurs, all blocks after the fork point are discarded. The node then rolls back all changes made by those blocks, such as spent outputs and account balances. It does this by iterating through the blocks in reverse order and rolling back each block's changes using the inverse of the block connections indicated in mapNextTx.

The mempool stored in src/txmempool.cpp must also be updated for reorgs. When a reorg occurs, the node calls removeForBlock() to remove all transactions included in the removed portion of the chain. It then calls UpdateTransactionsFromBlock() to process the transactions in the newly connected block. This may add new transactions to the mempool or remove others that have been included. Transactions with inputs spent in the new chain are removed. The node also updates transaction ancestor/descendant relationships and removes any transactions now considered expired.

Fee Estimation

References: src/wallet/feebumper.cpp, src/wallet/feebumper.h

The …/feebumper.cpp and …/feebumper.h files contain code related to estimating appropriate fees for wallet transactions. The feebumper namespace in feebumper.cpp contains the main logic for bumping transaction fees.

A key part of fee estimation is the EstimateFeeRate function, which calculates a new fee rate to use for the replacement transaction. It factors in node and wallet policies for minimum relay fees from …/policy.h.

The PreconditionChecks function ensures a transaction can be bumped by checking it has no descendants, is replaceable, and meets other criteria. The CheckFeeRate function validates that the provided fee rate is high enough based on these policies.

The CreateRateBumpTransaction function handles generating the replacement transaction. It creates the necessary inputs and outputs, calculates the new fee, and returns the result. SignTransaction then signs the transaction using the wallet.

The SignatureWeights class tracks the signature weights added during signature checking. Its AddSigWeight method accumulates these weights based on signature version. The SignatureWeightChecker overrides signature checking to also update the weights reference, allowing checking to occur while gathering this metadata needed for fee estimation.

CommitTransaction submits the signed transaction to the network, updating wallet metadata to mark the original transaction as replaced. This completes the fee bumping process.

Startup and Shutdown

References: src/init.h

The node::NodeContext represents the node being initialized and contains the state needed for initialization and shutdown. Initialization of the node context involves setting up core functionality in stages, with AppInit functions handling different stages.

AppInitMain performs the main initialization, taking a reference to the node::NodeContext and returning a boolean. It initializes logging, parameters, locks the data directory, initializes interfaces and starts background services.

The Shutdown function interrupts and stops threads managed by the node::NodeContext gracefully. ShutdownRequested can be used to check if a shutdown was requested on the context.

InitLogging configures logging based on command line arguments parsed by the ArgsManager. InitParameterInteraction allows changing parameters based on rules.

AppInitLockDataDirectory locks the data directory to prevent processes from writing to it concurrently. AppInitInterfaces initializes important interface pointers needed by other components.

StartIndexBackgroundSync starts regularly syncing indexes like the UTXO set in the background. This allows the node to catch up indexes like the UTXO database during initial block download.

Script Evaluation

References: src/script/interpreter.cpp, src/script/interpreter.h

The core script evaluation logic is implemented in the EvalScript function defined in …/interpreter.cpp. This function takes a script and stack as input, and iterates through each opcode, executing the appropriate logic to modify the stack. Opcodes related to signature validation invoke the GenericTransactionSignatureChecker class, defined in …/interpreter.h, to perform the cryptographic checks against a signature hash.

GenericTransactionSignatureChecker supports both ECDSA and Schnorr signatures. For ECDSA it implements the CheckECDSASignature() method, which calls the lower level VerifyECDSASignature() to do the cryptographic verification. For Schnorr it provides CheckSchnorrSignature() and VerifySchnorrSignature().

The signature hash used for these checks is computed by SignatureHash(), also defined in …/interpreter.h. This function implements the signature hash algorithm according to the active transaction version and standard.

Witness program scripts are executed using the ExecuteWitnessScript() function in …/interpreter.cpp. This enforces clean stack rules and size limits appropriate for witness execution.

Taproot support involves new primitives like ComputeTapleafHash() and ComputeTaprootMerkleRoot() defined in …/interpreter.h. These implement the cryptographic functions needed for Taproot and Tapscript.

Logging and Debugging

References: src/logging.h, src/logging.cpp, src/test/util/setup_common.cpp

The Logger class in …/logging.h and …/logging.cpp handles logging functionality for Bitcoin Core. It defines log categories to group different parts of the software, such as NET and RPC. Log levels like Trace and Debug determine the severity of messages.

The Logger manages writing log messages to various outputs like the console and files. It formats messages with timestamps and thread information. Global logging functions like LogPrint and LogPrintf provide an interface to the logger while enforcing category and level checks.

Categories are represented by LogFlags bitflags and can be individually enabled or disabled. The default log level and per-category levels filter messages. The logger buffers messages before logging starts to prevent loss.

RAII locking with StdLockGuard ensures the logging methods remain thread-safe. Callbacks can receive messages. Log configuration includes options like the filename and whether to print timestamps.

The BasicTestingSetup class in …/setup_common.cpp initializes dependencies for tests like logging and the node context. ChainTestingSetup builds on this for the chainstate, block and transaction handling.

Utilities create, process and access blocks and transactions. For example, CreateBlock() generates blocks and CreateValidTransaction() creates signed outputs between transactions. The TestChain100Setup provides a test chain of 100 blocks.

LoadVerifyActivateChainstate() loads, verifies and activates the best chainstate. Classes like ChainstateManager manage specific testing components. The file provides all core utilities and setup for generating test data and running node/wallet tests.

Release Notes

References: doc/release-notes

Release notes serve as a historical record of changes between versions of software. For open source projects, release notes help both users and developers understand what has changed in each release. This allows users to smoothly upgrade their installations and test for regressions, while informing developers of relevant codebase updates.

The Bitcoin Core project maintains release notes in the …/release-notes directory to document changes between versions of the Bitcoin Core software. Each version has its own markdown file containing information about that release. The notes provide a high-level overview of new features, improvements, and bugfixes. For significant changes, more technical details may be included to help explain relevant business logic updates.

Some key aspects covered in release notes include:

  • Instructions for upgrading existing nodes to the new version, including any special steps needed like database conversions.

  • Compatibility information like supported operating systems and dependencies.

  • A notable changes section highlighting major user-facing additions or alterations.

  • A detailed changelog categorizing specific fixes and changes to code, tests, documentation and other areas.

  • Credits to all contributors to recognize their work.

Release notes serve as both a historical record for users and a transparency mechanism for developers. By documenting the evolution of the software over time, they help new users and developers understand the project at a high level without needing deep implementation knowledge.

Testing Infrastructure

References: src/test, test, src/bench

The Bitcoin test infrastructure comprehensively validates core functionality through unit tests, functional tests, fuzz testing, and benchmarks. Unit tests focus on low-level components in isolation using the Boost test framework. They are defined in files matching the name of the code under test, such as …/arith_uint256_tests.cpp.

Functional tests use the BitcoinTestFramework to launch real Bitcoin nodes and interact with them via RPC and P2P. Defined in …/functional, they connect nodes with NodeConn and call RPCs with rpc() to test full node functionality. test_framework.py provides common test utilities and fixtures.

Fuzz testing is implemented in …/fuzz by generating randomized inputs to exercise code paths on binaries like bitcoind. Targets are implemented in files like block.cpp and functions are used to register targets.

Benchmarks in …/bench measure performance of key components. They subclass benchmark and override Run() to contain the code to benchmark. The BENCHMARK macro registers benchmarks, while bench.cpp runs them based on command line arguments. Benchmarks are organized by subsystem into directories like addrman.cpp.

The test_runner.py file in …/util loads tests from JSON and executes them using bctest(). It validates outputs and return codes. Common utilities are provided in this directory. Static analysis linting is implemented in …/lint.

Unit Tests

References: src/test/arith_uint256_tests.cpp, src/test/bloom_tests.cpp

Unit tests provide thorough, automated testing of low-level components. The …/test directory contains unit tests that validate the basic functionality of key classes, functions and algorithms within Bitcoin Core.

Some important unit test files include …/arith_uint256_tests.cpp, which tests arithmetic operations on large unsigned integers using the arith_uint256 class. It exercises construction, comparison operations, bitwise logic, shifts, and all arithmetic functions like addition, subtraction and multiplication. Edge cases are covered to detect any off-by-one errors.

Another key file is …/bloom_tests.cpp, which tests bloom filters implemented by the CBloomFilter class. It validates basic filter creation, insertion of data like transactions and keys, matching against inserted elements, serialization, and integration with CMerkleBlock objects. Tests cover different construction parameters and ensure expected elements are included or excluded by the filter.

Unit tests provide an important part of the testing infrastructure by validating low-level components in isolation at a small scale. They ensure basic logic and data structures like arith_uint256 and CBloomFilter function as intended before being used in larger modules. The tests are run automatically on every code change to catch any regressions.

Functional Tests

References: test/functional, test/functional/test_framework

The BitcoinTestFramework class provides functionality for writing end-to-end node and wallet tests. It handles initializing TestNode objects which represent individual nodes, and connecting them in various configurations. Methods like start_nodes(), connect_nodes(), and generate() allow controlling the node network programmatically.

The TestNode class encapsulates an individual node. It has methods for interacting with nodes over RPC like getblockchaininfo() and getmempoolentry().

The P2PInterface class allows testing P2P messaging between nodes. It inherits from P2PConnection and adds callbacks for message handling. The on_version(), on_inv(), etc. callbacks can be overridden to customize message processing.

The MiniWallet class in wallet.py supports basic transaction creation without a real wallet. Its create_self_transfer() method signs and sends simple self-transfer transactions.

The test_framework module contains common assertions, logging functionality, and other utilities that can be reused across tests. Functions like connect_nodes(), disconnect_nodes(), and sync_blocks() help control the node network state.

Overall, the framework provides the necessary infrastructure and abstractions to set up nodes, control their behavior, generate/sign transactions and blocks, and write tests that simulate real-world workflows between nodes, wallets, and over P2P. Tests can focus on implementing specific scenarios without needing to reimplement node control, block/transaction generation, or common assertions.

Fuzz Testing

References: src/test/fuzz

Fuzz testing involves generating randomized or unexpected inputs to software in order to uncover bugs and vulnerabilities. The main goal of fuzz testing is to find crashes in the code that could potentially be exploited.

The …/fuzz directory contains fuzz testing targets for various parts of the Bitcoin Core codebase. These targets are implemented using the libFuzzer framework. libFuzzer is a mutation-based fuzzing engine that generates random but valid inputs to exercise the code. It is able to find crashes and bugs that other techniques may miss by systematically exploring the input space.

Many files in this directory define FUZZ_TARGET functions that take a FuzzedDataProvider object as input. The FuzzedDataProvider breaks up a byte array into smaller pieces that can be consumed as strings, integers, or other data types. This allows libFuzzer to generate randomized test inputs for targets by mutating the initial seed corpus.

Targets deserialize and validate structures like transactions, blocks, and other primitives from the fuzzed inputs. They also call functions on objects constructed from the inputs. Any exceptions or crashes during this process would indicate potential bugs or vulnerabilities. By generating many random inputs, fuzz testing aims to improve robustness.

Some key files related to fuzz testing include:

The FuzzedDataProvider class is central to fuzz testing, as it allows libFuzzer to break input corpora into randomized test cases. Targets leverage this to fully exercise code across many random inputs.

Benchmarks

References: src/bench

The core functionality benchmarked in the Bitcoin Core code relates to measuring the performance of different components under various conditions. Key aspects include:

  • The benchmark namespace and BenchRunner class provide the framework for registering, running, and collecting results from benchmarks.

  • Benchmarks are implemented by subclassing benchmark::Bench and overriding the Run() method to contain the code being tested.

  • The BENCHMARK() macro registers benchmarks with the framework by name, priority level, and function.

  • Command line arguments in benchmark::Args control execution settings like filtering, output options, and input sizes.

  • Benchmarks are organized into subdirectories based on the system component or algorithm being tested.

  • Benchmarks aim to realistically simulate component usage by generating test data, dependencies, and full workloads where possible.

  • Performance is measured by batching operations and running benchmarks repeatedly to filter out noise.

  • Results include statistics like minimum, maximum, median times and are output to files.

Some key implementation details:

The BenchRunner::RunAll() method iterates the registered benchmark map. It filters based on regex and priority level specified in Args. Each matching benchmark is run by constructing a Bench object, which precisely measures timing and complexity. Output is generated using templates.

Benchmarks simulate component usage by setting up realistic dependencies. For example, the block_assemble.cpp benchmark populates the mock mempool with 1000 transactions before assembling a block, to test performance under a full workload.

Performance of algorithms is tested under varying conditions, like different input sizes or parameters. For example, poly1305.cpp benchmarks Poly1305 on 64B, 256B and 1MB inputs to cover a range of real-world sizes.

Test Utilities

References: src/test/util, test/util

The Bitcoin test suite provides several common utilities that allow tests to be written in a clean, modular way. Key utilities include tools for initializing testing environments, generating randomized test data, and validating test results.

The BasicTestingSetup class defined in …/setup_common.h and implemented in …/setup_common.cpp handles basic node configuration that many tests require. Its constructor initializes logging, command line arguments, and a NodeContext instance via functions like InitSignatureCache(). This provides a common initialization that individual test fixtures can build upon via inheritance.

The CMutableTransaction class in …/transaction_utils.h represents a transaction that can be modified. Functions like CreateBlock() and CreateValidTransaction() in …/mining.cpp construct CMutableTransaction objects deterministically, simplifying transaction generation for tests.

The CCoinsViewCache class referenced in files like …/coins.cpp acts as an in-memory unspent transaction output set cache on top of a backing store. The AddTestCoin() function in …/coins.cpp populates it with fake coins by creating Coin objects and adding them to the cache, providing a standardized way for tests to work with UTXO sets.

The ConnmanTestMsg class defined in …/net.cpp simulates P2P network connections and message passing without an actual network. Methods like Handshake(), ReceiveMsgFrom(), and FlushSendBuffer() allow rigorous testing and validation of P2P protocol implementations by programmatically creating nodes and simulating communications between them.

The DebugLogHelper class defined in …/logging.cpp searches debug logs for a specific message string. It registers a callback with the logging system to search logs for a given message and assert that message is seen before the test finishes.

Linting

References: test/lint

This section of the wiki covers the static analysis lints implemented in the test suite. Static analysis lints involve programmatically analyzing code to check for bugs, code quality issues, and conventions without executing the code.

The …/main.rs file provides a framework for running various static analysis checks, or "lints", on the Bitcoin codebase. It defines a LintResult type representing success or failure.

Some key linting checks implemented include:

  • External Python scripts like …/check-doc.py and …/all-lint.py are executed to check for documentation, code style issues, dead code, and other linting rules.

The main() function defines a list of lint checks to run. Any lint failures cause the test suite to fail via the exit code.

This provides an automated framework to run static analysis checks on the codebase and standardize the results. External scripts implement the specific linting logic.

Continuous Integration

References: ci

The Bitcoin Core project leverages a robust continuous integration (CI) system to automate building, testing, and validating code changes. This helps ensure all code passes automated tests and meets quality standards before being merged. The core CI infrastructure resides in the ci directory and its subdirectories.

Some of the key aspects of the CI system include:

  • The …/test directory contains scripts for setting up isolated build environments, installing dependencies, compiling Bitcoin Core, and executing the test suites. These scripts are run by the top-level orchestration scripts.

  • Environment setup scripts like 00_setup_env.sh initialize important variables to configure the build containers, packages, test behaviors, and other parameters. This ensures a reproducible environment.

  • Dependency installation is handled in 01_base_install.sh. It downloads dependencies like compilers and libraries once and caches them for future reuse.

  • Tests are run inside Docker containers by 02_run_container.sh to isolate the build from the host system. It compiles a container image defined by the Dockerfile.

  • 03_test_script.sh executes the actual test logic, running unit tests, fuzz tests, functional tests, and other validation methods like sanitizers.

  • Orchestration scripts like test_run_all.sh coordinate running all the individual test scripts to fully validate changes across different configurations.

  • Utilities like ci/retry provide robustness by retrying commands that may fail transiently, avoiding errors due to temporary outages.

  • Configuration is managed through environment variables, allowing flexible control of build options, dependencies, test behaviors and more.

  • Caches like ccache speed up rebuilds by reusing object files where possible. Dependency caches also improve performance.

The automated CI/CD pipeline is crucial to Bitcoin Core's quality assurance and enables identifying issues early before code is merged. It helps maintain the project's high standards for stability, security and functionality.