FUND MainNet Network Upgrade & Migration Guide

Codegnosis
6 min readMay 25, 2022

The most recent version of this guide is available here: https://github.com/unification-com/mainnet/blob/master/042_upgrade.md

Upgrading from und v1.4.8 (Cosmos SDK 0.38) to und v1.5.1 (Cosmos SDK 0.42)

There are two main stages to the upgrade process:

  1. Pre-upgrade preparation — this should be carried out by Validator nodes, and specifies when the chain should halt. It should be executed before the proposed halt time.
  2. Upgrade Process (after halt time) — the actual software upgrade and chain state migration process. The process is different depending on whether a Validator node is being upgraded, or a non-Validator node (for example a Sentry), and is executed after the proposed halt time is reached (and the chain has halted), but before the new proposed genesis time. See the respective sections below.

This guide assumes the und binary is running with systemd and that the user logging in via SSH is a sudoer. It also assumes that UND_HOME directory is the default $HOME/.und_mainchain directory path.

Only Validator nodes will be responsible for setting the chain halt-time and running the initial upgrade, genesis export and genesis migration to the Cosmos SDK 0.42 format. Other nodes (for example sentries, RPCs etc.) can then follow suit and use the new genesis.json generated by the validator(s).

Governance Proposal

See MainNet Proposal #5

Full proposal text: https://ipfs.io/ipfs/QmZRGGNWCQGu6tVDYc713rqAKvTH7Qv3FQxneNQe77uKqB?filename=FUND_MainNet_2_Upgrade_Proposal.md

MainNet Halt time: 1654686000 (Wed Jun 08 2022 11:00:00 GMT+0000)
New MainNet Chain ID: FUND-MainNet-2
New MainNet Genesis time: 2022–06–08T14:00:00Z

1. Validator Nodes: Pre-upgrade preparation

Validators will need to configure the halt-time of their node BEFORE the proposed halt time

Important: ONLY VALIDATOR NODES NEED TO SET THIS. Do not set for sentries/RPCs etc. as they will need to propagate the final block signatures to other validators (for example, those behind sentry nodes)

Edit your validator’s app.toml to define the specified halt-time. This is a Unix epoch timestamp, and will tell the node to gracefully stop producing blocks at the specified time.

Important: Ensure only the halt-time is set. Do not set halt-height.

nano $HOME/.und_mainchain/config/app.toml

Set as per the governance proposal & save:

halt-time=1654686000

Next, in a separate terminal, monitor your node’s logs to see when it’s “safe” to restart

sudo journalctl -u und -f

wait for a "Committed Block" message, then restart to load the config updates:

sudo systemctl restart und

2. Validator Nodes: Upgrade Process (after halt time)

This should be executed before any upgrades to Sentries etc.

The commands below assume $HOME/.und_mainchain is used for the und home directory. Replace as appropriate for your node installation.

Near the specified halt-time, monitor your node's logs with either journalctl or by querying the service's status, for example

sudo journalctl -u und -f

or by periodically checking the service status

sudo systemctl status und -l -n 100

Note: you may need to increase the number of lines output -n 100 in the above command.

Just after the specified halt-time you should see the message:

halting node per configuration               module=main height=0 time=1654686000
Committed state module=state height=NNNNNN txs=0 appHash=ABC123...

(where NNNNNN is the last committed block number)

It is likely you will also see many warnings after this log output, such as

Connection failed @ recvRoutine (reading byte) ...
Stopping peer for error ...
...

This means that other nodes in your node’s address book have likely (and hopefully) also halted at the specified time.

Once your node has cleanly halted as per the log message above, stop the systemd service:

sudo systemctl stop und

BACKUP, BACKUP, BACKUP .und_mainchain, for example:

rsync -av $HOME/.und_mainchain/ $HOME/und_mainchain_BAK/

Or

cd $HOME/.und_mainchain
tar -cpzf $HOME/und_mainchain.backup.tar.gz *

In the event the upgrade fails, the chain can be rolled back to und v1.4.8 and the old chain data.

Depending on the method used to backup, and your node’s configuration, the backup may consume 90Gb — 150Gb of additional space. This can be deleted or moved offline once the upgrade is successful and the new chain is running.

BACKUP your node & validator keys, preferably offline.

download und v1.4.9. This can be to a temporary location, e.g. $HOME/und_1.4.9 as it will only be used for exporting the state to a new genesis:

cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.4.9/und_v1.4.9_linux_x86_64.tar.gz
tar -zxvf und_v1.4.9_linux_x86_64.tar.gz
mv und und_1.4.9

Export the current chain state to the new genesis.
You must use und v1.4.9 to execute this:

$HOME/und_1.4.9 export --for-zero-height --home=$HOME/.und_mainchain > $HOME/v038_exported_state.json

backup the old und and undcli 1.4.8 binaries, e.g.

mkdir -p $HOME/UND_BIN_BACKUP
cp /usr/local/bin/und $HOME/UND_BIN_BACKUP/und_1.4.8
cp /usr/local/bin/undcli $HOME/UND_BIN_BACKUP/undcli_1.4.8

IMPORTANT undcli is no longer required since all the old undcli commands are not integrated into the single und binary. As such, it should be removed if it installed in a common location because it will not be compatible with und v1.5.1, for example:

sudo rm /usr/local/bin/undcli

download und v1.5.1 and install in /usr/local/bin or your preferred location, for example:

cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.5.1/und_v1.5.1_linux_x86_64.tar.gz
tar -zxvf und_v1.5.1_linux_x86_64.tar.gz
sudo mv und /usr/local/bin/und

IMPORTANT: All commands from here require und v1.5.1

Ensure the installed und is v1.5.1

/usr/local/bin/und version --log_level=""

Mirgate the exported genesis using the und v1.5.1 binary:

/usr/local/bin/und migrate $HOME/v038_exported_state.json --chain-id=FUND-MainNet-2 --genesis-time="2022-06-08T14:00:00Z" --log_level="" > $HOME/new_v042_genesis.json

IMPORTANT Set the --log_level="" in the command. Without it, the binary will add log info to the migrated genesis file!

Validate the migrated genesis:

/usr/local/bin/und validate-genesis --log_level="" $HOME/new_v042_genesis.json

Generate a checksum and cross-reference with other operators in Discord/Telegram:

jq -S -c -M '' $HOME/new_v042_genesis.json | sha256sum
[SHA256SUM_PLACEHOLDER] -

Note: jq is required to be installed for the above command.

Reset the chain data for your node:

/usr/local/bin/und unsafe-reset-all --log_level="" --home=$HOME/.und_mainchain

Copy the new genesis file to replace the old version in $HOME/.und_mainchain/config:

cp $HOME/new_v042_genesis.json $HOME/.und_mainchain/config/genesis.json

Download the new 0.42 formatted app.toml and replace your old version in $HOME/.und_mainchain/config:

curl https://raw.githubusercontent.com/unification-com/mainnet/master/latest/042_app.toml > $HOME/.und_mainchain/config/app.toml

Modify as required:

nano $HOME/.und_mainchain/config/app.toml

Change the log_level in $HOME/.und_mainchain/config/config.toml, as the variable format has changed in Cosmos SDK 0.42.x:

nano $HOME/.und_mainchain/config/config.toml

and set:

log_level = "info"

Note: You may also want to check the changes to config.toml, and apply any to your node. The biggest changes are the addition of the State Sync Configuration Options section. The version included in this repo is for reference only and should not be used to overwrite your existing config.toml!

Finally, start the node!

sudo systemctl start und
sudo journalctl -u und -f

IMPORTANT: The chain will not start producing blocks until both the new genesis time is reached, AND enough Validator nodes are back online. Once this happens, blocks will begin from block #1. This may not occur until after the specified genesis time, as validators and sentries come back online and connect. Keep in contact with other validator operators in Discord/Telegram to keep up to date with each other’s upgrade status!

3. Sentries, RPCs, Seeds etc.

Once Validators have been configured and started, the genesis file can be copied over to other nodes. There is no need to export and migrate the genesis on sentry/RPC/seed nodes.

Stop the node:

sudo systemctl stop und

Backup at least one full $HOME/.und_mainchain if you have multiple sentries.

Backup the config directory for all sentry/rpc/seed nodes:

cd $HOME/.und_mainchain/config
tar -cpzf $HOME/und_mainchain.backup.tar.gz *

backup the old und and undcli v1.4.8 binaries, and remove the old installed undcli e.g.

mkdir -p $HOME/UND_BIN_BACKUP
cp /usr/local/bin/und $HOME/UND_BIN_BACKUP/und_1.4.8
cp /usr/local/bin/undcli $HOME/UND_BIN_BACKUP/undcli_1.4.8
sudo rm /usr/local/bin/undcli

There is no need to download und v1.4.9 - it will not be used for non-validator nodes.

Download und v1.5.1 and install it in /usr/local/bin or your preferred location:

cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.5.1/und_v1.5.1_linux_x86_64.tar.gz
tar -zxvf und_v1.5.1_linux_x86_64.tar.gz
sudo mv und /usr/local/bin/und

IMPORTANT: All commands from here require und v1.5.1

Ensure it’s v1.5.1

/usr/local/bin/und version --log_level=""

Reset the chain data for your node:

/usr/local/bin/und unsafe-reset-all --log_level="" --home=$HOME/.und_mainchain

Transfer/copy the new genesis.json, for example from your Validator node or Github once it's published, and overwrite the old $HOME/.und_mainchain/config/genesis.json

Download and configure the app.toml for Cosmos 0.42 chains.

curl https://raw.githubusercontent.com/unification-com/mainnet/master/latest/042_app.toml > $HOME/.und_mainchain/config/app.toml

It is recommended to set some values for the [state-sync] section at the end of app.toml, which will benefit the network by speeding up state syncing for new nodes joining the network. Assuming pruning = "default", some suggested values are:

snapshot-interval = 500
snapshot-keep-recent = 4

Change the log_level in config.toml

nano $HOME/.und_mainchain/config/config.toml

and set:

log_level = "info"

Finally, start the node!

sudo systemctl start und
sudo journalctl -u und -f

Important Links

GitHub: https://github.com/unification-com

Telegram: https://t.me/unificationfoundation

Discord: https://discord.com/channels/725618617525207042

Website: www.unification.com

--

--