Auto Restart / Auto Upgrade
◽ Introduction
Bottom line: A node that disconnects from the cluster(s) is a node that isn’t earning. As Node Operators, we must remain vigilant to ensure our nodes do not disconnect from the cluster.
Auto Restart will assist us with this task.
◽ What is Auto Restart?
The auto_restart feature was designed to detect if your node is not properly connected to the Hypergraph and/or metagraphs.
Auto Restart command reference can be found here.
If an auto_restart might interfere with another nodectl command issued by the Node Operator, nodectl will automatically stop the auto_restart service and restart it upon completion of the requested task.
◽ What is Auto Upgrade?
The auto_upgrade command is an extension of the auto_restart feature, working in conjunction with it.
If nodectl detects that your Node is not properly connected to the Hypergraph and/or metagraphs, as specified in the configuration, it will not only attempt to bring your node back online but also ensure that the version of Tessellation is up to date by automatically upgrading your Node.
Note
auto_upgrade requires that auto_restart is also enabled.
◽ Passphrase Requirement
Auto Restart must be able to unlock your p12 keystore in order to properly restart or upgrade your node.
◽ How to enable Auto Restart
There are two ways to enable auto_restart and auto_upgrade.
◽ Recommend Method
You may use the auto_restart quick start guide if you do not want to review the details of this step-by-step guide.
Steps to configure auto_restart in the configuration:
Connect to your Node
Enter the configuration using the configure command.
Suggestion
In the beginning, it is better not to enter into advanced mode. This will offer you more detailed explanations of each aspect of the configuration.
You can just hit the enter key to accept the default n to stay in detailed mode.
sudo nodectl configure
= CONSTELLATION NETWORK HYPERGRAPH =
= NODECTL =
= CONFIGURATION TOOL =
========================================
Code Name: Princess Warrior
Welcome to the nodectl configuration tool.
This feature of nodectl will help you initialize a new configuration or update/edit an existing configuration file.
nodectl will attempt to migrate/integrate your configurations changes in order to ensure a smooth transition and operations of your Node via nodectl.
Detailed Mode: will walk you through all steps/questions; with detailed explanations of each element of the configuration.
Advanced Mode: will be non-verbose, with no walk through explanations, only necessary questions.
The configuration tool does only a limited amount of data type or value verification. After the configuration tool creates a new configuration or edits an existing configuration, it will attempt to verify the end resulting configuration.
You can also choose the -a option at the command line to enter advanced mode directly.
Continue in advanced mode? [n]: n
- Edit the existing configuration using the E key (do not hit enter).
= MAIN MENU =
========================================
N)ew Configuration
E)dit Existing Configuration
Q)uit
KEY PRESS an option
- Press any key to acknowledge the configuration backup.
A previous cn-config.yaml was found on the system.
In the event the backup directory was not found, a backup was created in the existing directory. The location is shown below.
/var/tessellation/backups/cn-config_2023-05-31-23:44:16Z
Press any key to continue
- We will choose the R key to enter the
auto_restartsection.
= OPTIONS MENU =
========================================
E) Edit Individual Profile Sections
A) Append New Profile to Existing
G) Global P12 Section
R) Auto Restart Section
M) Main Menu
Q) Quit
KEY PRESS an option
- Choose y to enable auto_restart
- Choose y to enable auto_upgrade (optional)
- Choose y to enable on_boot (optional)
- Confirm the choices you made.
= CONFIRM VALUES =
========================================
If you reached this confirmation unexpectedly ,from the input [above] you may have hit <enter> along with your option; therefore, choosing the default. You can choose n here and reenter the correct value.
enable auto restart: y
enable auto upgrade: y
enable on boot: y
Please confirm values are as requested: [y]:
- Your configuration file will be properly rebuilt, and you will be offered the ability to review the configuration file.
building [cn-config.yaml] file................. complete
Review the created configuration? [y]:
- After you review (or skip) the configuration review, your configuration will be validated again, and final instructions will be shown.
WARNING auto_restart was modified in the configuration. The configurator will not disable/enable any instances of auto_restart automatically.
To enable issue : sudo nodectl auto_restart enable
To disable issue: sudo nodectl auto_restart disable
Press any key to continue
You can either enable or disable via the commands referenced.
Warning
nodectl is not a program that continuously runs on your Node (not including auto_restart which does run as a service.) On a reboot of your server or VPS, your auto_restart service will not enable until a nodectl command is executed which will trigger the auto_restart service to engage if enabled.
- Exit out of the configuration feature by pressing the q key.
= OPTIONS MENU =
========================================
E) Edit Individual Profile Sections
A) Append New Profile to Existing
G) Global P12 Section
R) Auto Restart Section
M) Main Menu
Q) Quit
KEY PRESS an option
nodeadmin@Constellation-Node:~#
◽ Manual Method
warning
This method should only be used in more specifically cases where using the configuration is not an option.
This method can also be used in conjunction with having auto_restart enabled in the configuration.
Steps to manually enable auto restart
sudo nodectl auto_restart enable
sudo nodectl auto_restart disable
sudo nodectl auto_restart restart
sudo nodectl auto_restart enable --auto_upgrade
see auto_restart or auto_upgrade command reference for more details.
◽ Deep Dive
nodectl will processing each profile in its own thread (i/o).
◽ Enablement / Disablement
The Node Operator should use nodectl's configure command to persist the enablement or disablement the auto_restart feature. This cannot be accomplished by manually enabling this feature.
◽ What does nodectl monitor?
Monitoring Activities
| Monitor Type | Description | Mechanism |
|---|---|---|
| Restart | If the cluster has a restart. | session |
| Upgrade | If the cluster is identified to have been upgraded. | session |
| Minority Fork | If the node forks away from the majority fork. | various reasons |
| Majority Fork | If the node is no longer on the proper cluster. | session |
| Stuck in State | If the node is stuck in a non Ready state. Currently includes: WaitingForDownload, SessionStarted, Observing, and WaitingForReady. | timers |
| Rollback | If the cluster has been rolled back for any particular reason. | session |
◽ Timing?
An auto_restart may take up to ~18 minutes to complete.
These long executions occur because the node detects one or both profiles as down and first restarts the profile related to the global Hypergraph. nodectl will then attempt to bring up any metagraph profiles. To avoid timing conflicts with other nodes that may also have auto_restart enabled, random timers are applied throughout the restart process.
nodectl will need to properly link your metagraph profile to the global Hypergraph.
It is important to understand this is a background and unattended process, the delay is created on purpose.
It is recommended by the developers to link a metagraph (that requires this type of setup) through your node's own global Hypergraph connection.
If you are using auto_restart please remember if you are physically monitoring your node while it is enabled, you need to exercise patience to allow it to figure out how to get back online by itself as necessary.
An auto_restart may take up to approximately 18 minutes to complete.
These extended execution times occur because the node first detects one or both profiles as down and then restarts the profile related to the global Hypergraph. nodectl will subsequently attempt to bring up any metagraph profiles. To avoid timing conflicts with other nodes that may also have auto_restart enabled, random timers are applied throughout the restart process.
nodectl needs to properly link your metagraph profile to the Global Hypergraph.
It’s important to understand that this is a background and unattended process, and the delay is intentional.
Developers recommend linking a metagraph that requires this setup through your node’s own Global Hypergraph connection.
If you are using auto_restart, please remember that if you are actively monitoring your node while it is enabled, you need to be patient and allow it to figure out how to get back online independently as needed.
◽ Manual Interoperability
Forcing a manual restart (or any service affecting command) will disable auto_restart. If enabled in the configuration, nodectl will attempt to re-enable auto_restart after any command that requires it to be temporarily disabled. If the Node Operator does not have auto_restart enabled in the configuration, it will not re-enable after-the-fact.
In order to avoid duplicate or unwanted behavior such as your node restarting when you do not want it started, the auto_restart feature will automatically disable if you attempt to issue any command that manipulates the services.
Automatic Disablement Commands
| Command | Command | Command |
|---|---|---|
| leave | stop | start |
| join | restart | upgrade |
| restart_only | show_restart | nodectl_upgrade |
| display_snapshot_chain | execute_starchiver |