The PolPatrol tool¶

Hand-crafted tests¶

PolPatrol’s hand-crafted tests generate extrinsics that test the following list of modules:

• Balances: Multiple transfer scenarios (successful/unsuccessful) covering all 3x functions, including calls from root origin.
• Claim: One example is presented.
• Slots: Auctioning off parachain slots for a certain lease duration. A bid to an auction is placed. The Adder parachain runtime is used as an example to onboard a new parachain and set its head data.
• ImOnline: A heartbeat is crafted.
• FinalityTracker: A successful and a non-successful is available.
• Sudo: The sudo module is a helper module to dispatch root origin calls in runtimes to be tested. This is supported, in the default genesis, the root key is set to Alice.

Following these examples, it is easy to add additional tests for these or new modules. A minblock can be specified per extrinsic to ensure that the extrinsic is not applied before a certain block number.

Unchecked extrinsics are fully supported. There are two types of Unchecked extrinsics, signed extrinsics and inherent extrinsics. Inherent extrinsic can only be applied by the block author during the generation of the block, signed extrinsics are similar to transactions. Some extrinsics require root_origin. If the relay-chain runtime under test features the Sudo module, the Sudo module can be used to dispatch with root_origin. The pre-configured user Alice has been initialized as the Sudo key (see the initialization section below).

If the Sudo module is not available in the runtime under test (as we assume to be the case for releases of the Polkadot relay-chain runtime in the future), a solution to circumvent this might be to use wasm2wat and to deactivate the check of ensure_root_origin.

Fuzzing¶

PolPatrol supports two types of fuzzing for input generation:

1. Fuzz calldata: In this mode, PolPatrol generates completely random calldata and passes it to the Runtime Entry to be fuzzed.
2. Fuzz components: In this mode, PolPatrol generates random headers and blocks with the expected components (headers, extrinsics, and blocks) but it randomly generates the individual components (i.e., it will generate a completely random header for the block). Extrinsics are fetched randomly from the extrinsics pool, this ensures meaningful input.

Users can specify as a command-line argument the number of fuzzing inputs that will be generated by PolPatrol. By default, this number is set to zero.

We note that the fuzzing tests are expected to trigger errors. To this end, PolPatrol warns the user in case a random test input results in a successful call.

Initialization¶

The initial state of the environment is defined by the provided genesis storage file. Currently, PolPatrol uses a slightly modified version of the Kusama-cc2 genesis state.

At the initial state, there are five accounts: Alice, Bob, Charlie, Dave, and Eve. Accounts Alice, Bob, and Charlie have positive balances at the initial state, while accounts Dave and Eve have their balances set to zero.

Block generation and execution¶

To execute the tests, PolPatrol first loads the relay-chain runtime under test. Then, it queries the version of the loaded runtime, which is needed when generating the extrinsics. Further, it queries the metadata to know which extrinsics are available at the runtime (this can be used to skip the hand-crafted extrinsics that are not supported by the runtime).

Next, PolPatrol generates new blocks in a loop, including a fixed number of extrinsics in each one, until no more extrinsics are left to execute.

PolPatrol uses two execution paths:

1. One execution path where the new block is generated. The new block is initialized using Core_initialize_block()

   • Inherent extrinsics are generated using BlockBuilder_inherent_extrinsics
   • Extrinsics to be included are fetched
   • Extrinsics are applied using BlockBuilder_apply_extrinsic
   • The new block is finalized using BlockBuilder_finalize_block

2. A second execution path where the newly generated block is applied against the previous state using Core_execute_block.

Each execution path has its own execution environment with its own storage. At the end of a block, the roots of both storages are compared to ensure that they do not diverge.

Notes on test execution:

• Test execution stops upon detection of storage root mismatch (between the 2x execution paths: aggregation of the block vs execution of the block) at the end of a block, an appropriate error message tells the user to fix the affected extrinsic.

• If the execution of an extrinsic fails with Ok(Err(*)), then this leads to a storage root mismatch while failures with Ok(Ok(Fail)) (e.g., due to an insufficient balance or lack of permission) are do not result in a storage root mismatch.

• For the current Kusama-cc2 runtime, the module OnlyStakingandClaims is active. This means the current code for the runtimes contains a hardcoded limitation to fail all other calls, see /runtime/src/lib.rs inside the Polkadot repository:

  fn validate(&self, _: &Self::AccountId, call: &Self::Call, _: DispatchInfo, _: usize)
    -> TransactionValidity
  {
    match call {
      Call::Staking(_) | Call::Claims(_) | Call::Sudo(_) | Call::Session(_) =>
        Ok(Default::default()),
          => Err(InvalidTransaction::Custom(ValidityError::NoPermission.into()).into()),
    }
  }
  

  To compile the Runtime for proper testing, these checks need to be commented out and instead a return statement Ok(Default::default()) must be added. This allows to call any of the include module successfully.