FUND MainNet Network Upgrade & Migration Guide
The most recent version of this guide is available here: https://github.com/unification-com/mainnet/blob/master/042_upgrade.md
Upgrading from und v1.4.8 (Cosmos SDK 0.38) to und v1.5.1 (Cosmos SDK 0.42)
There are two main stages to the upgrade process:
- Pre-upgrade preparation — this should be carried out by Validator nodes, and specifies when the chain should halt. It should be executed before the proposed halt time.
- Upgrade Process (after halt time) — the actual software upgrade and chain state migration process. The process is different depending on whether a Validator node is being upgraded, or a non-Validator node (for example a Sentry), and is executed after the proposed halt time is reached (and the chain has halted), but before the new proposed genesis time. See the respective sections below.
This guide assumes the und
binary is running with systemd
and that the user logging in via SSH is a sudoer
. It also assumes that UND_HOME
directory is the default $HOME/.und_mainchain
directory path.
Only Validator nodes will be responsible for setting the chain halt-time
and running the initial upgrade, genesis export and genesis migration to the Cosmos SDK 0.42 format. Other nodes (for example sentries, RPCs etc.) can then follow suit and use the new genesis.json
generated by the validator(s).
Governance Proposal
See MainNet Proposal #5
Full proposal text: https://ipfs.io/ipfs/QmZRGGNWCQGu6tVDYc713rqAKvTH7Qv3FQxneNQe77uKqB?filename=FUND_MainNet_2_Upgrade_Proposal.md
MainNet Halt time: 1654686000 (Wed Jun 08 2022 11:00:00 GMT+0000)
New MainNet Chain ID: FUND-MainNet-2
New MainNet Genesis time: 2022–06–08T14:00:00Z
1. Validator Nodes: Pre-upgrade preparation
Validators will need to configure the halt-time
of their node BEFORE the proposed halt time
Important: ONLY VALIDATOR NODES NEED TO SET THIS. Do not set for sentries/RPCs etc. as they will need to propagate the final block signatures to other validators (for example, those behind sentry nodes)
Edit your validator’s app.toml
to define the specified halt-time
. This is a Unix epoch timestamp, and will tell the node to gracefully stop producing blocks at the specified time.
Important: Ensure only the halt-time
is set. Do not set halt-height
.
nano $HOME/.und_mainchain/config/app.toml
Set as per the governance proposal & save:
halt-time=1654686000
Next, in a separate terminal, monitor your node’s logs to see when it’s “safe” to restart
sudo journalctl -u und -f
wait for a "Committed Block"
message, then restart to load the config updates:
sudo systemctl restart und
2. Validator Nodes: Upgrade Process (after halt time)
This should be executed before any upgrades to Sentries etc.
The commands below assume $HOME/.und_mainchain
is used for the und
home directory. Replace as appropriate for your node installation.
Near the specified halt-time
, monitor your node's logs with either journalctl
or by querying the service's status, for example
sudo journalctl -u und -f
or by periodically checking the service status
sudo systemctl status und -l -n 100
Note: you may need to increase the number of lines output -n 100
in the above command.
Just after the specified halt-time
you should see the message:
halting node per configuration module=main height=0 time=1654686000
Committed state module=state height=NNNNNN txs=0 appHash=ABC123...
(where NNNNNN
is the last committed block number)
It is likely you will also see many warnings after this log output, such as
Connection failed @ recvRoutine (reading byte) ...
Stopping peer for error ...
...
This means that other nodes in your node’s address book have likely (and hopefully) also halted at the specified time.
Once your node has cleanly halted as per the log message above, stop the systemd
service:
sudo systemctl stop und
BACKUP, BACKUP, BACKUP .und_mainchain
, for example:
rsync -av $HOME/.und_mainchain/ $HOME/und_mainchain_BAK/
Or
cd $HOME/.und_mainchain
tar -cpzf $HOME/und_mainchain.backup.tar.gz *
In the event the upgrade fails, the chain can be rolled back to und
v1.4.8 and the old chain data.
Depending on the method used to backup, and your node’s configuration, the backup may consume 90Gb — 150Gb of additional space. This can be deleted or moved offline once the upgrade is successful and the new chain is running.
BACKUP your node & validator keys, preferably offline.
download und
v1.4.9. This can be to a temporary location, e.g. $HOME/und_1.4.9
as it will only be used for exporting the state to a new genesis:
cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.4.9/und_v1.4.9_linux_x86_64.tar.gz
tar -zxvf und_v1.4.9_linux_x86_64.tar.gz
mv und und_1.4.9
Export the current chain state to the new genesis.
You must use und
v1.4.9 to execute this:
$HOME/und_1.4.9 export --for-zero-height --home=$HOME/.und_mainchain > $HOME/v038_exported_state.json
backup the old und
and undcli
1.4.8 binaries, e.g.
mkdir -p $HOME/UND_BIN_BACKUP
cp /usr/local/bin/und $HOME/UND_BIN_BACKUP/und_1.4.8
cp /usr/local/bin/undcli $HOME/UND_BIN_BACKUP/undcli_1.4.8
IMPORTANT undcli
is no longer required since all the old undcli
commands are not integrated into the single und
binary. As such, it should be removed if it installed in a common location because it will not be compatible with und
v1.5.1, for example:
sudo rm /usr/local/bin/undcli
download und
v1.5.1 and install in /usr/local/bin
or your preferred location, for example:
cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.5.1/und_v1.5.1_linux_x86_64.tar.gz
tar -zxvf und_v1.5.1_linux_x86_64.tar.gz
sudo mv und /usr/local/bin/und
IMPORTANT: All commands from here require und
v1.5.1
Ensure the installed und
is v1.5.1
/usr/local/bin/und version --log_level=""
Mirgate the exported genesis using the und
v1.5.1 binary:
/usr/local/bin/und migrate $HOME/v038_exported_state.json --chain-id=FUND-MainNet-2 --genesis-time="2022-06-08T14:00:00Z" --log_level="" > $HOME/new_v042_genesis.json
IMPORTANT Set the --log_level=""
in the command. Without it, the binary will add log info to the migrated genesis file!
Validate the migrated genesis:
/usr/local/bin/und validate-genesis --log_level="" $HOME/new_v042_genesis.json
Generate a checksum and cross-reference with other operators in Discord/Telegram:
jq -S -c -M '' $HOME/new_v042_genesis.json | sha256sum
[SHA256SUM_PLACEHOLDER] -
Note: jq
is required to be installed for the above command.
Reset the chain data for your node:
/usr/local/bin/und unsafe-reset-all --log_level="" --home=$HOME/.und_mainchain
Copy the new genesis file to replace the old version in $HOME/.und_mainchain/config
:
cp $HOME/new_v042_genesis.json $HOME/.und_mainchain/config/genesis.json
Download the new 0.42 formatted app.toml
and replace your old version in $HOME/.und_mainchain/config
:
curl https://raw.githubusercontent.com/unification-com/mainnet/master/latest/042_app.toml > $HOME/.und_mainchain/config/app.toml
Modify as required:
nano $HOME/.und_mainchain/config/app.toml
Change the log_level
in $HOME/.und_mainchain/config/config.toml
, as the variable format has changed in Cosmos SDK 0.42.x:
nano $HOME/.und_mainchain/config/config.toml
and set:
log_level = "info"
Note: You may also want to check the changes to config.toml
, and apply any to your node. The biggest changes are the addition of the State Sync Configuration Options
section. The version included in this repo is for reference only and should not be used to overwrite your existing config.toml
!
Finally, start the node!
sudo systemctl start und
sudo journalctl -u und -f
IMPORTANT: The chain will not start producing blocks until both the new genesis time is reached, AND enough Validator nodes are back online. Once this happens, blocks will begin from block #1. This may not occur until after the specified genesis time, as validators and sentries come back online and connect. Keep in contact with other validator operators in Discord/Telegram to keep up to date with each other’s upgrade status!
3. Sentries, RPCs, Seeds etc.
Once Validators have been configured and started, the genesis file can be copied over to other nodes. There is no need to export and migrate the genesis on sentry/RPC/seed nodes.
Stop the node:
sudo systemctl stop und
Backup at least one full $HOME/.und_mainchain
if you have multiple sentries.
Backup the config directory for all sentry/rpc/seed nodes:
cd $HOME/.und_mainchain/config
tar -cpzf $HOME/und_mainchain.backup.tar.gz *
backup the old und
and undcli
v1.4.8 binaries, and remove the old installed undcli
e.g.
mkdir -p $HOME/UND_BIN_BACKUP
cp /usr/local/bin/und $HOME/UND_BIN_BACKUP/und_1.4.8
cp /usr/local/bin/undcli $HOME/UND_BIN_BACKUP/undcli_1.4.8
sudo rm /usr/local/bin/undcli
There is no need to download und
v1.4.9 - it will not be used for non-validator nodes.
Download und
v1.5.1 and install it in /usr/local/bin
or your preferred location:
cd $HOME
wget https://github.com/unification-com/mainchain/releases/download/1.5.1/und_v1.5.1_linux_x86_64.tar.gz
tar -zxvf und_v1.5.1_linux_x86_64.tar.gz
sudo mv und /usr/local/bin/und
IMPORTANT: All commands from here require und
v1.5.1
Ensure it’s v1.5.1
/usr/local/bin/und version --log_level=""
Reset the chain data for your node:
/usr/local/bin/und unsafe-reset-all --log_level="" --home=$HOME/.und_mainchain
Transfer/copy the new genesis.json
, for example from your Validator node or Github once it's published, and overwrite the old $HOME/.und_mainchain/config/genesis.json
Download and configure the app.toml
for Cosmos 0.42 chains.
curl https://raw.githubusercontent.com/unification-com/mainnet/master/latest/042_app.toml > $HOME/.und_mainchain/config/app.toml
It is recommended to set some values for the [state-sync]
section at the end of app.toml
, which will benefit the network by speeding up state syncing for new nodes joining the network. Assuming pruning = "default"
, some suggested values are:
snapshot-interval = 500
snapshot-keep-recent = 4
Change the log_level
in config.toml
nano $HOME/.und_mainchain/config/config.toml
and set:
log_level = "info"
Finally, start the node!
sudo systemctl start und
sudo journalctl -u und -f
Important Links
GitHub: https://github.com/unification-com
Telegram: https://t.me/unificationfoundation
Discord: https://discord.com/channels/725618617525207042
Website: www.unification.com