Skip to content

Commit

Permalink
Update Ledgerutil Docs
Browse files Browse the repository at this point in the history 
Signed-off-by: Julian Castrence <[email protected]>
  • Loading branch information
@jrc-ibm @denyeart
jrc-ibm authored and denyeart committed Jul 1, 2022
1 parent e235d41 commit 12a047a
Show file tree
Hide file tree
Showing 5 changed files with 304 additions and 32 deletions.
177 changes: 162 additions & 15 deletions docs/source/commands/ledgerutil.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,131 @@

# ledgerutil

The `ledgerutil compare` command allows administrators to compare channel snapshots from two different peers.
Although channel snapshots from the same height should be identical across peers, if the snapshot
files indicate different state hashes (as seen in `_snapshot_signable_metadata.json` file from each snapshot)
then it indicates that at least one of the peers state databases is not correct relative to the blockchain and
may have gotten corrupted.
The `ledgerutil` or ledger utility suite is a set of tools useful for troubleshooting instances on a Fabric network where the ledgers of peers on a channel have diverged. The tools can be used individually or sequentially in a variety of divergence scenarios with the goal of helping the user locate the initial cause of divergence.

The `ledgerutil compare` utility will output a set of JSON files if the snapshots are not identical to assist with troubleshooting in these situations.
Two output JSON files will include any key/value differences sorted by key (one for public key/value differences and one for private key/value differences),
another JSON file will include any key/value differences (public or private) sorted by block and transaction height so that you can identify the height where a divergence may have first occurred.
## Syntax

The output files may help an administrator to understand the scope of a state database issue and identify which keys are impacted.
Snapshots from additional peers can be compared to determine which peer has incorrect state.
The block and transaction height of the first difference can be used as a reference point when looking at the peer's logs to understand what may have happened to the state database at that time.
The `ledgerutil` command has two subcommands

## Syntax
* `compare`
* `identifytxs`

## compare

The `ledgerutil compare` command allows administrators to compare channel snapshots from two different peers. Although channel snapshots from the same height should be identical across peers, if the snapshot files indicate different state hashes (as seen in `_snapshot_signable_metadata.json` file from each snapshot) then it indicates that at least one of the peer's state databases is not correct relative to the blockchain and may have gotten corrupted.

The `ledgerutil compare` command will output a directory containing a set of JSON files if the snapshots are not identical to assist with troubleshooting in these situations. Two output JSON files will include any key/value differences sorted by key, one for public key/value differences and one for private key/value differences. A third JSON file will include any key/value differences (public or private) sorted by block and transaction height so that the user can identify the height where a divergence may have first occurred. The ordered key/value difference JSON file will contain the earliest n differences, where n is a user chosen number passed as a flag to the `ledgerutil compare` command or a default value of 10 when no flag is passed.

The output files may help an administrator to understand the scope of a state database issue and identify which keys are impacted. Snapshots from additional peers can be compared to determine which peer has the incorrect state. The block and transaction height of the first difference can be used as a reference point when looking at the peer's logs to understand what may have happened to the state database at that time. Below is an example of an ordered difference output JSON file:

```
{
"ledgerid":"mychannel",
"diffRecords":[
{
"namespace":"_lifecycle$$h_implicit_org_Org1MSP",
"key":"f0e4c2f76c58916ec258f246851bea091d14d4247a2fc3e18694461b1816e13b",
"hashed":true,
"snapshotrecord1":null,
"snapshotrecord2":{
"value":"0c52a3dbc7b322ff35728afdd691244cfc0fc9c4743c254b57059a2394e14daf",
"blockNum":1,
"txNum":0
}
},
{
"namespace":"_lifecycle$$h_implicit_org_Org1MSP",
"key":"e01e1c4304282cc5eda5d51c41795bbe49636fbf174514dbd4b98dc9b9ecf5da",
"hashed":true,
"snapshotrecord1":{
"value":"0c52a3dbc7b322ff35728afdd691244cfc0fc9c4743c254b57059a2394e14daf",
"blockNum":1,
"txNum":0
},
"snapshotrecord2":null
},
{
"namespace":"marbles",
"key":"marble1",
"hashed":false,
"snapshotrecord1":{
"value":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\"}"
"blockNum":3,
"txNum":0
},
"snapshotrecord2":{
"value":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"jerry\"}",
"blockNum":4,
"txNum":0
}
}
]
}
```

The field `ledgerid` indicates the name of the channel where the comparison took place. The field `diffRecords` provides a list of key/value differences found between the compared snapshots. The compare tool found 3 differences. The first diffRecord has a null value for the field `snapshotrecord1` indicating that the key is present in the second snapshot and missing in the first snapshot. The second diffRecord has a null value for the field `snapshotrecord2` indiciating the opposite scenario. Both diffRecords have a `hashed` value of true, indicating that they are private key/value differences. The third diffRecord contains data in both snapshot fields, indicating that the snapshots have divergent data for the same key. Further examination reveals the snapshots are in disagreement about the owner of the key `marbles marble1` and the height at which that value is set. The `hashed` field is set to false indicating this is a public key/value difference.

## identifytxs

The `ledgerutil` command has one subcommand
The `ledgerutil identifytxs` command allows administrators to identify lists of transactions that have written to a set of keys in a peer's local block store. The command takes a JSON file input containing a list of keys and outputs a JSON file per key, each file containing a list of transactions, if the available block range is valid. When troubleshooting a divergence, the command is most effective when paired with the `ledgerutil compare` command, by taking a `ledger compare` generated output JSON file of key differences as input, so that `ledgerutil identifytxs` can identify the transactions on the blockchain associated with the in doubt keys. The command does not necessarily have to be used in tandem with the `ledgerutil compare` command and can be used with a user generated JSON list of keys or an edited `ledgerutil compare` output JSON file to filter transactions in a more general approach.

* compare
As previously mentioned, the `ledgerutil identifytxs` command is designed to accept the output of the `ledgerutil compare` command, however the tool is compatible with any JSON file that is formatted the same way. This allows for users to perform a more direct troubleshooting approach in circumstances where they may be more informed on the cause of the divergence. The input JSON file should contain a `ledgerid` that directs the tool to the channel name it must conduct the search on and a `diffRecords` list providing a list of keys for the tool to search for within transactions. Below is an example of a valid input JSON file:

```
{
"ledgerid":"mychannel",
"diffRecords":[
{
"namespace":"marbles",
"key":"marble1",
"snapshotrecord1":{
"value":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\"}","blockNum":3,
"txNum":0
},
"snapshotrecord2":{
"value":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"jerry\"}","blockNum":4,
"txNum":0
}
}
{
"namespace":"marbles",
"key":"marble2"
}
]
}
```

Notice that in the above example for key `marbles marble2`, the fields for the snapshot records and the height of the key's found divergence which are present in output JSON files generated by the `ledgerutil compare` command are exempt. In this case, the entire available block store would be searched as there is no known height of divergence.

The output of the `ledgerutil identifytxs` command is a directory containing a set of JSON files, one per input key. Continuing the example above, if the example JSON was used as the input file for the `ledgerutil identifytxs` command, the resulting output would be a directory containing two JSON files named `txlist1.json` and `txlist2.json`. The output JSON files are named in the order they appear in the input JSON file, so `txlist1.json` would contain a list of transactions that wrote to the namespace key pairing `marbles marble1` and `txlist2.json` would contain a list of transactions that wrote to the namespace key pairing `marbles marble2`. Below is an example of the contents of `txlist1.json`:

```
{
"namespace":"marbles",
"key":"marble1",
"txs": [
{
"txid":"9ccb0d0bf19f143b29f17254364ccae987a8d89317f8e8dd81228762fef9da5f",
"blockNum":2,
"txNum":2,
"txValidationStatus":"VALID",
"keyWrite":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"john\"}"
},
{
"txid":"a67c735fa1ef3390199aa2669a4f8023ea469cfe213afebf1014e57bceaf0a57",
"blockNum":4,
"txNum":0,
"txValidationStatus":"VALID",
"keyWrite":"{\"docType\":\"marble\",\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\"}"
}
],
"blockStoreHeightSufficient":true
}
```

The above output JSON file indicates that for the key `marbles marble1`, two transactions were found in the available block store that wrote to `marbles marble1`, the first transaction occuring at block 2 transaction 2, the second occuring at block 4 transaction 0. The field `blockStoreHeightSufficient` is used to inform the user if the available block range was sufficient for a full search up to the given key's height. If true, the last available block in the block store was at least at the height of the key's height of divergence, indicating the block store height was sufficient. If false, the last available block in the block store was not at the height of the key's height of divergence, indicating there may be transactions relevant to the key that are not present. In the case of the example, `blockStoreHeightSufficient` indicates that the block store's block height is at least 4 and that any transactions past this height would be irrelevant for troubleshooting purposes. Since no height of divergence was provided for the key `marbles marble2`, the `blockStoreHeightSufficient` would default to true in the `txlist2.json` file and becomes a less useful point of troubleshooting information.

A block range is valid if the earliest available block in the local block store has a lower height than the height of the highest input key; if otherwise, a block store search for the input keys would be futile and the command throws an error. It is important to understand that, in cases where the earliest block available is greater than block 1 (which is typical for block stores of peers that have been bootstrapped by a snapshot), the output of the command may not be an exhaustive list of relevant transactions since the earliest blocks were not available to be searched. Further troubleshooting may be necessary in these circumstances.
## ledgerutil compare
```
usage: ledgerutil compare [<flags>] <snapshotPath1> <snapshotPath2>
Expand All @@ -47,6 +152,29 @@ Args:
<snapshotPath2> Second ledger snapshot directory.
```


## ledgerutil identifytxs
```
usage: ledgerutil identifytxs [<flags>] <snapshotDiffsPath> [<blockStorePath>]
Identify potentially divergent transactions.
Flags:
--help Show context-sensitive help (also try --help-long
and --help-man).
-o, --outputDir=OUTPUTDIR Location for identified transactions json results
output directory. Default is the current directory.
Args:
<snapshotDiffsPath> Path to json file containing list of target records to
search for in transactions. This is typically output from
ledgerutil compare.
[<blockStorePath>] Path to file system of target peer, used to access block
store. Defaults to '/var/hyperledger/production'.
IMPORTANT: If the configuration for target peer's file
system path was changed, the new path MUST be provided.
```

## Exit Status

### ledgerutil compare
Expand All @@ -55,6 +183,11 @@ Args:
- `2` if the snapshots are not identical
- `1` if an error occurs

### ledgerutil identifytxs

- `0` if the block store was successfully searched for transactions
- `1` if an error occurs or the block range is invalid

## Example Usage

### ledgerutil compare example
Expand All @@ -67,7 +200,6 @@ Here is an example of the `ledgerutil compare` command.
ledgerutil compare -o ./compare_output -f 10 ./peer0.org1.example.com/snapshots/completed/mychannel/5 ./peer1.org1.example.com/snapshots/completed/mychannel/5
Both snapshot public state and private state hashes were the same. No results were generated.
```
The response above indicates that the snapshots are identical. If the snapshots were not identical, the command results will indicate where the comparison output files are written, for example:
Expand All @@ -78,4 +210,19 @@ Here is an example of the `ledgerutil compare` command.
* Note that both snapshot locations must be accessible by the command, for example by mounting volumes from two different peers, or by copying the snapshots to a common location.
### ledgerutil identifytxs example
Here is an example of the `ledgerutil identifytxs` command.
* Identify relevant transactions in mychannel from a block store within an allowed block range. The input JSON file contains a list of keys to search for in the block store. The block range is from the earliest block in the block store to the tallest block height for any key in the input JSON file. The block store must at least have a height equal to the height of the tallest key in the input JSON file to be considered a valid block range.
```
ledgerutil identifytxs compare_output/mychannel_5_comparison/first_diffs_by_height.json -o identifytxs_output
Successfully ran identify transactions tool. Transactions were checked between blocks 1 and 4.
```
The response above indicates that the local block store was successfully searched. This means transactions within the block range that wrote to keys found in the output JSON of the compare command were identified. In the newly created directory identifytxs_output, a directory mychannel_identified_transactions was generated containing a JSON file of identified transactions for each key from the compare command JSON output.
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
2 changes: 1 addition & 1 deletion docs/source/docs_guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ To add a new CLI command, perform the following steps:
associated help text. See `internal/peer/version` for a simple example to get
started.
- Add a section for your CLI command in
`src/github.com/hyperledger/fabric/scripts/generateHelpDoc.sh`.
`src/github.com/hyperledger/fabric/scripts/help_docs.sh`.
- Create two new files under `/src/github.com/hyperledger/fabric/docs/wrappers`
with the associated content:
- `<command>_preamble.md` (Command name and syntax)
Expand Down
21 changes: 20 additions & 1 deletion docs/wrappers/ledgerutil_postscript.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@
- `2` if the snapshots are not identical
- `1` if an error occurs

### ledgerutil identifytxs

- `0` if the block store was successfully searched for transactions
- `1` if an error occurs or the block range is invalid

## Example Usage

### ledgerutil compare example
Expand All @@ -18,7 +23,6 @@ Here is an example of the `ledgerutil compare` command.
ledgerutil compare -o ./compare_output -f 10 ./peer0.org1.example.com/snapshots/completed/mychannel/5 ./peer1.org1.example.com/snapshots/completed/mychannel/5
Both snapshot public state and private state hashes were the same. No results were generated.
```
The response above indicates that the snapshots are identical. If the snapshots were not identical, the command results will indicate where the comparison output files are written, for example:
Expand All @@ -29,4 +33,19 @@ Here is an example of the `ledgerutil compare` command.
* Note that both snapshot locations must be accessible by the command, for example by mounting volumes from two different peers, or by copying the snapshots to a common location.
### ledgerutil identifytxs example
Here is an example of the `ledgerutil identifytxs` command.
* Identify relevant transactions in mychannel from a block store within an allowed block range. The input JSON file contains a list of keys to search for in the block store. The block range is from the earliest block in the block store to the tallest block height for any key in the input JSON file. The block store must at least have a height equal to the height of the tallest key in the input JSON file to be considered a valid block range.
```
ledgerutil identifytxs compare_output/mychannel_5_comparison/first_diffs_by_height.json -o identifytxs_output
Successfully ran identify transactions tool. Transactions were checked between blocks 1 and 4.
```
The response above indicates that the local block store was successfully searched. This means transactions within the block range that wrote to keys found in the output JSON of the compare command were identified. In the newly created directory identifytxs_output, a directory mychannel_identified_transactions was generated containing a JSON file of identified transactions for each key from the compare command JSON output.
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
Loading

0 comments on commit 12a047a

Please sign in to comment.