Skip to main content

upgrading with nodectl

Description

This document will show you; through step-by-step instructions, how to upgrade your node to the latest version of Tessellation or nodectl.

Tessellation is the code name for the protocol that runs on your node. It is the guts of how your node is able to operate on the Hypergraph or metagraph.

nodectl is a utility that runs on your node and helps you automate some of the more complex processes that Tessellation requires to run efficiently.

Quick Start

In order to upgrade your node.

  1. Issue the command: sudo nodectl upgrade
  2. Follow the prompts.

In order to upgrade nodectl.

  1. Issue the command: sudo nodectl upgrade_nodectl
  2. Follow the prompts.

Details of the UPGRADE process

Prerequisite

Installation

This document assumes that you have nodectl with Tessellation already running on your node.

If this is not the case, please review the installation guide Getting Started: to determine the type of installation required.

Upgrade

Upgrade nodectl

Upgrade the nodectl utility.

Upgrade Tessellation

Upgrade Tessellation using nodectl.

Upgrading with nodectl

There are 3 upgrade mechanisms regarding nodectl.

  1. Upgrade the nodectl utility to a new version. (binary)
  2. Upgrade your node to handle new features of nodectl, or modify some elements to make your node more efficient when working with the nodectl utility.
  3. Upgrade Tessellation and any elements of the VPS that may require modification to work with the Tessellation protocol.

Mechanisms 2 and 3 may include updating features of the VPS to facilitate functionality of your node.

Using nodectl's built-in upgrade command, you do not have to worry about the differences mentioned when upgrading nodectl. nodectl is smart enough to guide you through the upgrade process. It will know if you need any extra steps to complete.

These steps include:

  • Is it safe to upgrade from the current version of nodectl your node is running to the latest version?
    • There is an upgrade path that might need to be taken; dependent on, how long you have waited to upgrade your node.
  • Does your node require any extra upgrade steps to complete the nodectl upgrade for effective operation.

Upgrade to new version of nodectl

nodectl is simply a single binary files compiled to work on 1 of either 2 Linux architectures. That is to say, it is single file that you can simply download from the Internet and execute on your system.

warning

nodectl has 2 separate binary files. Both work only on Linux Debian based distributions

  • x86_64
  • arm64

In the unlikely event you need to do a manual upgrade of nodectl, it will be necessary to make sure the binary is executable (This is an advanced system management process). We will cover manual downloads below.

Upgrade detection

nodectl is smart enough to attempt to determine if there is an upgrade available. By reaching out to an external source (scrapping the nodectl repository on GitHub ) nodectl will check for the latest upgrade.

As an example (the list command is unrelated to this document's purpose), a sudo nodectl list command was issued, and a new version on nodectl was detected.

The full output of the list command was omitted.

nodeadmin@Constellation-Node:~# sudo nodectl list
A new version of nodectl was detected:
v2.7.1
To upgrade issue: sudo nodectl upgrade_nodectl

In the above output, it shows us that our node is not up-to-date and the latest version available is v2.7.1.

Upgrade nodectl via nodectl

If nodectl is already installed on your system, we can issue the upgrade_nodectl command to attempt to upgrade.

sudo nodectl upgrade_nodectl

When we execute the upgrade_nodectl command; in this example below, the auto_restart feature (The auto_restart command is unrelated this document's purpose.) was enabled. Since nodectl cannot upgrade while the auto_restart feature is actively running, nodectl will disable the feature auto_magically for us.

nodectl will detect that there is a new version and ask us if we are sure we want to continue? We can say y here.

nodeadmin@Constellation-Node:~# sudo nodectl upgrade_nodectl
FOUND auto_restart instance.
AutoRestart service with pid [3101736] ........ disabled
Auto Restart will reengage at completion of requested task
A *new* version of nodectl was detected:.........
v2.7.1
To upgrade issue: sudo nodectl upgrade_nodectl
WARNING This will upgrade mainnet
You are currently on: MAINNET
version: v2.7.0
available: v2.7.1
Are you sure you want to continue? [n]: y

nodectl will begin the upgrade of nodectl. In the background through automation nodectl will:

  1. Download the necessary binary perspective GitHub repository.
  2. Place the binary in the correct location on your VPS/node.
  3. Update the permissions of the nodectl binary as required.
  4. Determine if a node upgrade is necessary.

Architecture

nodectl will detect the architecture of our VPS, in this example x86_64 ( ( most common ).

Because nodectl cannot upgrade while it is running, nodectl will:

  • Exit itself before upgrading.
  • Upgrade.
    • nodectl will create a temporary bash script.
    • execute the script.
  • Inform us whether or not we need to upgrade anything necessary for Tessellation or the Linux distribution to operate effectively with nodectl.
  • Re-enable auto_restart if enabled.

In this example:

  • We do not need to upgrade our node.
  • We have auto_restart enabled in the configuration.

nodectl will let us know, and request we press enter to continue without an upgrade and then restart the auto_restart feature.

Upgrading nodectl version from v2.7.0 to v2.7.1

Detected architecture: x86_64
WARNING nodectl will exit to upgrade.
Please be patient and allow the upgrade to complete before
continuing to work.

COMPLETED! nodectl upgraded to v2.7.1
VERSION MAJOR MINOR PATCH
v2.7.1 2 7 1

This version of nodectl DOES NOT require an upgrade be performed
Press [ENTER] to continue...

The auto_restart service will restart.

This version of nodectl DOES NOT require an upgrade be performed
Press [ENTER] to continue...

node restart service started...

In the event our node requires an internal upgrade of the node components, for various reasons including:

  • Tessellation changes
  • VPS changes
  • nodectl changes

The output will look different, as shown below

Upgrading nodectl version from v2.7.0 to v2.7.1

Detected architecture: x86_64
WARNING nodectl will exit to upgrade.
Please be patient and allow the upgrade to complete before
continuing to work.

COMPLETED! nodectl upgraded to v2.7.1
VERSION MAJOR MINOR PATCH
v2.7.1 2 7 1

This version of nodectl requires an upgrade be performed
on your node.
Press Y then [ENTER] to upgrade or N then [ENTER] to cancel:

It is highly recommended to upgrade your node when requested. Failure to do so may result in undesirable results or failures.

Manual Installation

Follow the release notes instructions for the release you desire to install

nodectl releases

Upgrade Node to work with new nodectl features

It is very simple to upgrade your node with new nodectl features. We simply execute the upgrade command and follow the prompts.

sudo nodectl upgrade
Keep In Mind

If you are upgrading nodectl via the sudo nodectl upgrade command and not Tessellation, when requested to enter in the version of Tessellation you want to upgrade to, you will choose the same version. You should be able to simple hit the enter when nodectl requests the Tessellation versioning.

------ * Handle Node Versioning * ------

The following version is the latest ........... v1.11.0
The following version is running currently .... v1.11.0
Please enter version to upgrade to.........[v1.11.0] :

Since the process of an upgrade is exactly the same as the process necessary to upgrade Tessellation, we can skip to the Upgrade Tessellation section to see the step-by-step process.

Upgrade Tessellation

Introduction

nodectl's upgrade command serves 2 purposes simultaneously.

  1. Upgrade Tessellation.
  • Including any VPS (bare metal or container) background adds, changes, or enhancements that relate to Tessellation.
  1. Upgrade the VPS (bare metal or container) background adds, changes, or enhancements that may or may not relate to Tessellation ( may be related to the distribution or nodectl instead ).

Start Upgrade

From the remote login on our node. We can issue the upgrade command.

sudo nodectl upgrade
nodeadmin@Constellation-Node:~# sudo nodectl upgrade
Side Note

If the -ni (non-interactive) switch is included when the upgrade was executed, unless there is an issue that warrants nodectl to disengage non-interactive mode, you will not be prompted to answer any questions, instead off the defaults will be used.

sudo nodectl upgrade -ni

non-interactive if passphrase is removed from the configuration.

sudo nodectl upgrade -ni --pass <my_passphrase>

This document will assume that we are running the upgrade in interactive mode.

Confirm Upgrade

After execution of the sudo nodectl upgrade command, the screen will clear and you will be requested to confirm the upgrade you requested.

In this example, the auto restart feature was enabled. In order to upgrade, we will need to disable this feature prior to upgrading. If auto restart is running, it will conflict with the requests to start and stop services. Don't worry, nodectl will disable and re-enable the auto_restart feature for us.

We can choose y here.

========================================
= CONSTELLATION NETWORK HYPERGRAPH     =
= UPGRADE REQUEST                      =
= TESSELLATION VALIDATOR NODE          =
========================================
Code Name: Princess Warrior

FOUND auto_restart instance.
AutoRestart service with pid [3181577] ........ disabled
Auto Restart will reengage at completion of requested task
Terminating auto_restart ...................... complete
Are you sure you want to continue this upgrade? [n]: y

Handle OS System Upgrades

nodectl will begin the automated process of upgrading, we only need to sit back and watch.

nodectl will:

  • Make sure we are allowed to upgrade.
  • Identify our external IP address.
  • Update the system package list on the Debian distribution.
---- * Handle OS System Upgrades * -----

Check permissions & versioning ................ complete
Obtaining External IP ......................... 113.113.113.113
Updating the Debian OS system ................. complete

Verify Node Upgrade

nodectl will review the associated p12 file on your system and will identify the node id.

No user intervention is required.

------- * Verify Node Upgrade * --------

Obtaining node ID from p12 [global] ........... f1322....1fc84

Handle Node Versioning

nodectl will check the difference in versioning between what is currently running on your node and what version is known as the latest available version of Tessellation.

caution

In the event that you are only upgrading nodectl and not Tessellation, you will keep the same version by simply hitting enter to accept the default.

In the below example, nodectl identified that we are running version v1.10.0 on our node; however, v1.11.0 is the latest.

nodectl will auto-magically select the latest version for us, so we can just hit the enter key here.

danger

Entering an invalid version may result in an inoperable node. In the event this happens, you can restart the upgrade in an attempt to correct the issue.

------ * Handle Node Versioning * ------

The following version is the latest ........... v1.10.0
The following version is running currently .... v1.11.0
Please enter version to upgrade to.........[v1.11.0] :

Upgrade in progress

nodectl will now take over and start the upgrade process. You are free to sit back, relax and watch the progress.

Later in the upgrade ( unless the -ni switch was chosen ) you will be prompted to answer some questions related to the upgrade.

We recommend you don't leave it unattended to avoid timeouts, missed possible errors, or user interactive prompts that will require your attention.

nodectl will take the node off the cluster (HyperGraph and metagraph) in preparation for upgrade.

In the example below, you will see that nodectl identified dag-l0 and dag-l1 profiles configured on your node. It will gracefully attempt to remove itself from the clusters configured by these profiles.

-------- * Take Node Offline * ---------

Leaving the cluster for profile ............... dag-l1
Leaving the cluster for profile ............... dag-l0
Pausing: 6 of 30 seconds to allow node to gracefully leave
Out of Cluster

nodectl confirms our node is off the clusters

It will also identify the service name (Debian OS level) node_l0 and node_l1 that are running our profiles and stop them for us.

cluster status [dag-l1] ....................... out of cluster
cluster status [dag-l0] ....................... out of cluster
Stop request initiated [node_l1] .............. running
Stop request initiated [node_l0] .............. running
Stop request initiated [node_l1] .............. complete
Stop request initiated [node_l0] .............. complete

nodectl will issue a status command against the node to confirm the results of the stoppage.

info

Formatting of the window example is misrepresented due to formatting. The display on your node will be better represented.

SERVICE            JOIN STATE        PROFILE
inactive (dead)    ApiNotReady       dag-l1
PUBLIC API TCP     P2P API TCP       CLI API TCP
9010               9011              9012
CURRENT SESSION    FOUND SESSION     ON NETWORK
1683294942915      SessionNotFound   False

SERVICE            JOIN STATE        PROFILE
inactive (dead)    ApiNotReady       dag-l0
PUBLIC API TCP     P2P API TCP       CLI API TCP
9000               9001              9002
CURRENT SESSION    FOUND SESSION     ON NETWORK
1454859585788      SessionNotFound   False

Node Internal Configuration

nodectl will begin working on your your VPS elements and your node's elements, as necessary.

Note

The elements of the node that are setup here may change from version to version to adhere to necessities of proper and efficient node operation.

In the below example, nodectl is reviewing version 1 settings to confirm they are not enabled, this may be removed in future versions or keep in place to make sure all proper node operation is completed. These elements may have been already completed in a previous version or not applicable for your version; moreover, we should not be concerned.

--- * Node Internal Configuration * ----

Verifying Node directory setup ................ complete
Updating swapfile settings .................... skipped
NOTE: For partial or skipped elements, see the logs for details.
Removing v1.12.0 [service] files............... complete
Removing v1.12.0 [bash] files.................. complete
Building v2.0.0 Services Files ................ complete
In the event that the configuration yaml services changed nodectl will attempt to clean up old service files.
Skipping clean nothing to remove | file count: 0 files [0B]
Clean up config yaml changes v2.0.0 ........... complete

Clean up backups

It important to keep our node hard drive space clean of unnecessary files. nodectl will now offer you the opportunity to clean out your backups.

caution

You should review your backups to make sure you do NOT delete files you may have as a backup but do NOT want to delete.

Advanced users may want to transfer these files to a secondary backup directory to keep them safe.

You should have a backup of the following files in a secure location off of you node. These files contain passphrases and critical wallet contents which contain your passphrase.

  • p12 file
  • cn-config.yaml

The answer to this question, is up to you, in the example we said y.

-------- * Clean up backups * ---------

remove? /var/tessellation/backups/.bashrc_2023-0x-0x-14:42:14Z
remove? /var/tessellation/backups/.bashrc_2023-0x-0x-12:13:15Z
remove? /var/tessellation/backups/cn-node.2023-0x-xx-21:42:19Zbackup
disk space to be recovered: 69 KB
file count: 3
Are you sure you want to clear the selected backups? [n]: y

Clean up uploads

Similar to the backups you may have some files that were created in order to upload for diagnostics, logging, taxes, etc. We can clean up these files as well.

In the event that your node has files located in this special directory, you will be given a list of the files that will be removed and a confirmation prompt.

In the below example, the uploads directory was found to be empty, so the step was skipped

--------- * Clean up uploads * ---------

Skipping clean nothing to remove | file count: 0 files [0B]
Cleaning logs from [uploads] > 7 days ......... complete

Clean up logs

Similar to the backups and uploads, the logs directory can become large related to log files that are being built up and not removed.

In the below example, some of the log files were omitted with [...] representing files that were built up. You will notice Hypergraph layer0 and metagraph layer1 logs being represented.

---------- * Clean up logs * -----------

remove? [layer0] /var/tessellation/dag-l0/logs/archived/app.2023-05-02.0.log.gz
remove? [layer0] /var/tessellation/dag-l0/logs/archived/http.2023-05-02.0.log.gz
remove? [layer0] /var/tessellation/dag-l0/logs/archived/http.2023-05-01.5.log.gz
remove? [layer0] /var/tessellation/dag-l0/logs/archived/http.2023-05-03.2.log.gz
[...]
remove? [layer1] /var/tessellation/dag-l1/logs/archived/http.2023-04-30.2.log.gz
remove? [layer1] /var/tessellation/dag-l1/logs/archived/http.2023-05-02.3.log.gz
remove? [layer1] /var/tessellation/dag-l1/logs/archived/http.2023-05-01.1.log.gz
remove? [layer1] /var/tessellation/dag-l1/logs/archived/transactions.2023-04-30.0.log.gz
remove? [layer1] /var/tessellation/dag-l1/logs/archived/http.2023-05-01.3.log.gz
[...]
remove? [layer1] /var/tessellation/dag-l1/logs/json_logs/json.log
disk space to be recovered: 1 GB
file count: 65
Are you sure you want to clear the selected logs? [n]: y

Clean up snapshots

note

Due to the addition of incremental snapshots, this process may soon be deprecated.

Older snapshots are safe to remove from the node to clear up disk space. The snapshots data will over time cause your /var partition on your Debian system to fill up. If this happens, you will be faced with unknown or undesirable results both with your node's operations and nodectl's ability to load and run.

Here we can safely clear out our older snapshots.

-------- * Clean up snapshots * --------

snapshot directories can be large, please wait patiently...
disk space to be recovered: 5 GB
file count: 83179
Are you sure you want to clear the selected snapshots? [n]: y

Allow snapshot removal

nodectl will begin the process of removing the snapshots. This is done in a file by file basis to avoid timeouts and false-positive (perception of freezing due to background processes). Please have patience during this step.

-------- * Clean up snapshots * --------

snapshot directories can be large, snapshot removal will take a lot of time
please wait patiently...
removing [ 2696 ] of [ 83179 ]

once complete, output similar to the below example, should follow ...

disk space recovered: 5 GB
file count: 83179
cleaning logs from [snapshots] > 30 days ...... complete

Update packages and seed lists

nodectl will pull down the necessary packages that will upgrade your node to the latest versions.

We will also update any access permission lists (seed lists) that need to match for proper authentication to the Hypergraph and metagraphs.

note

If there is not a seed list present for a particular profile, the fetch will be disabled and skipped. You have the ability to configure this within the nodectl configuration file.

sudo nodectl configure

You can sit back and relax as nodectl continues to upgrade your node for you.

--------- * Handle Packages * ----------

Download [v2.0.0-alpha.5] Constellation Network Tessellation Binaries
Fetch Tessellation binary [cl-keytool.jar] .... complete
Fetch Tessellation binary [cl-node.jar] ....... complete
Fetch Tessellation binary [cl-dag-l1.jar] ..... complete
Fetch Tessellation binary [cl-wallet.jar] ..... complete
Fetching cluster seed file [testnet->dag-l0] .. complete
Fetching cluster seed file [testnet->dag-l1] .. disabled/skipped

Bring Node Back Online

nodectl will bring your service back online for you.

------ * Bring Node Back Online * ------

Reload the node's services .................... complete
Start request initiated [node_l0] ............. running
building environment .......................... complete
Updating services file ........................ complete
Start request initiated [node_l0] ............. complete

Then it will verify the status and display for you.

SERVICE             JOIN STATE        PROFILE
active (running)    ReadyToJoin       dag-l0
PUBLIC API TCP      P2P API TCP       CLI API TCP
9000                9001              9002
CURRENT SESSION     FOUND SESSION     ON NETWORK
1683294932039       SessionNotFound   ReadyToJoin

It is important to note here that we only brought back up our layer0 service. In this example the profile called dag-l0 is started.

A special recommended feature of nodectl is coordination to allow your node to proper join:

  • The layer0 network cluster joins first.
  • Join consensus for that layer.
  • Wait for Ready state (properly joined).
  • Allow the required layer1 connection to be created by linking through your node's layer0 network.

This is recommended because it provides a consistent and reliable layer0 to layer1 link for your node to function as efficiently as possible between clusters.

We will find our node is ReadytoJoin.

Checking for [ReadyToJoin] state............... ReadyToJoin
Please wait while [dag-l0] attempts to join the network.

Re-join the Network

The configured layer0 profile will rejoin the network. In this case the profile dag-l0 is configured as the layer0 and will attempt to join.

In this upgrade (document/manual) we did not choose the -w (watch) mode. This creates an upgrade that is less verbose, and saves time by not forcing the Node Operator to wait for all peer to peer connections to be established, instead once the node reaches a state where it is able to participate on the network, nodectl will skip watching for the remaining peers to connect and simply and safely continue the upgrade process.

It is necessary that all peers on the network learn about all other peers on the network via the gossip mechanisms in the Constellation Network's DAG protocol setup. However, it is not important to wait for all connections to be established, to complete the upgrade.

Below we will see that our node reached WaitingForReady while it was connected to 36 out of 48 known nodes on the network. nodectl continued the upgrade process...

---------- * Joining dag-l0 * ----------

Reviewing [dag-l0] ............................ ReadyToJoin
Preparing to join cluster [dag-l0] ............
Max Timer 300 seconds
Peers: 48 Connected: 36 State: WaitingForReady Timer: 20
Join process complete ......................... done
ok that peer count < cluster peer count
watch mode was not chosen by upgrade.

Checking status [dag-l0] ......................

SERVICE             JOIN STATE        PROFILE
active (running)    Ready             dag-l0
PUBLIC API TCP      P2P API TCP       CLI API TCP
9000                9001              9002
CURRENT SESSION     FOUND SESSION     ON NETWORK
1683294932039       1683294932039     True

Metagraph services layer1

nodectl will now start up your layer1 service which was configured in a profile called dag-l1.

Start request initiated [node_l1] ............. running
building environment .......................... complete
Updating services file ........................ complete
Start request initiated [node_l1] ............. complete

Check the status of dag-l1.

Fetching Status [dag-l1] ......................
SERVICE             JOIN STATE        PROFILE
active (running)    ReadyToJoin       dag-l1
PUBLIC API TCP      P2P API TCP       CLI API TCP
9010                9011              9012
CURRENT SESSION     FOUND SESSION     ON NETWORK
1683294942915       SessionNotFound   ReadyToJoin

We are in the ReadyToJoin state. Will have to now wait for dag-l0 profile, that is our layer0 profile to be in Ready state.

Checking for [ReadyToJoin] state............... ReadyToJoin
Please wait while [dag-l1] attempts to join the network.
NOTE: Layer1 networks will not join the Hypergraph until its Layer0 linked profile changes to Ready state, this could take up to a two minutes.

Since the layer0 was already in Ready state (meaning that it is properly and effectively participating on the HyperGraph). We can now link our Layer1 through our Layer0 to transfer data securely and properly between layers.

If our node is not properly in Ready state on layer0, nodectl will patiently wait for this condition to be met, updating the Node Operator through the command prompts, as it continues to check.

Joining next layer

nodectl will now attempt to join the dag-l1.

As mentioned above, in this upgrade (document/manual) we did not choose the -w (watch) mode. nodectl will not wait for all peers to connect, as it is not necessary to complete the upgrade.

---------- * Joining dag-l1 * ----------

Reviewing [dag-l1] ............................ ReadyToJoin
Waiting on profile [dag-l0] state to be [Ready]
before initiating Hypergraph join.
Current Found State [dag-l0] .................. Ready
Preparing to join cluster [dag-l1] ............
Max Timer 300 seconds
Peers: 49 Connected: 2 State: Ready Timer: 0
Join process complete ......................... done
ok that peer count < cluster peer count
watch mode was not chosen by upgrade.

nodectl will check the status of your node.

Checking status [dag-l1] ......................
SERVICE             JOIN STATE        PROFILE
active (running)    Ready             dag-l1
PUBLIC API TCP      P2P API TCP       CLI API TCP
9010                9011              9012
CURRENT SESSION     FOUND SESSION     ON NETWORK
1683294942915       1683294942915     True

Complete the upgrade

nodectl will complete the process.

dag-l1 upgrade process completed!
dag-l0 upgrade process completed!
Upgrade has completed

Total upgrade time: 4.865 minutes

Restart auto restart feature

Since the auto restart feature; in this example, was enabled, the auto restart feature will re-enable at the end of the upgrade for us.

node restart service started...

Upgrade Complete

Congratulations, you have completed a successful upgrade.