What Should I do When the Database Node is Down?

When database node is DOWN, troubleshoot using the following checklist. Step Task Results 1 Check whether your database is UP. $ admintools -t db_status -s…

OpenText profile picture
OpenText

June 14, 20182 minute read

Shot of two businesswomen using a digital tablet together during a collaboration at work

When database node is DOWN, troubleshoot using the following checklist.

Step Task Results
1 Check whether your database is UP.
$ admintools -t db_status -s UP
If the database is UP, go to Step 2.

If the database if not UP, restart your database.
$ admintools -t start_db -d <Database_name> -p <Database_password>

If the database starts, the checklist is complete.

If the database does not start, see the Database Process Not Starting checklist.

2 Identify all the DOWN nodes.
=> SELECT node_name, node_address, node_state FROM nodes WHERE node_state = 'DOWN';
Upon identification of all the DOWN node/s, proceed to Step 3.
3 Check whether you can establish a connection with the DOWN nodes using SSH.
$ ssh dbadmin @<nodedown_ip>
If you can SSH into the node, restart Vertica process on the DOWN node.
$ admintools -t restart_node -d <database_name> -s <node_host_name or IP>

  • If the restart was successful, the checklist is complete.
  • If the restart failed, go to Step 4.

If you cannot SSH into the node, contact your system administrator to find if it is a port issue or a network issue.

4 To find reasons for restart failure, on the DOWN node tail startup.log. A snippet of the tail startup.log is as follows:
$ tail -f catalog-path/database-name/v_database-name_node_catalog/startup.log
{
"node" : "v_cdmt0_node0001",
"stage" : "Database Halted",
"text" : Data consistency problems found; Check that all file systems are properly mounted.
Also, the --force option can be used to delete corrupted data.
"timestamp" : "2016-07-31 18:17:04.122"
}
The log results shows the latest state of the DOWN node. Proceed to Step 5 to see these stages.
5 If the startup.log…

a. Remains in the
Waiting for cluster invitestage.

See the Spread Debugging Checklist.
b. Remains in the Recovery stage. See the Node Recovery Checklist.
c. Shows an error message, Data inconsistency problems found, restart the node with the force option.
$ admintools -t restart_node -d Database_name -s node_name --force
Upon restart the checklist is complete.
d. Shows no new data in the startup.log after last restart, check dbLog file for errors. If you cannot resolve errors, contact Vertica Support.
e. Shows Shutdown Complete but the node is still DOWN.
tail vertica.log and look for <ERROR> and <PANIC>.
Contact Vertica Support with PANIC report, ErrorReport.txt, and scrutinize.

Share this post

Share this post to x. Share to linkedin. Mail to
OpenText avatar image

OpenText

OpenText, The Information Company, enables organizations to gain insight through market-leading information management solutions, powered by OpenText Cloud Editions.

See all posts

More from the author

Manutan combines digital services with the human touch to delight customers

Manutan combines digital services with the human touch to delight customers

At Manutan, we equip businesses and communities with the products and services they require to succeed. Headquartered in France, our company has three divisions, serving…

4 minute read

Reaching new markets in Europe and beyond

Reaching new markets in Europe and beyond

How information management specialists at One Fox slashed time to market for innovative products with OpenText Cloud Platform Services At One Fox, we’ve driven some…

4 minute read

SoluSoft helps government agencies tackle fraud faster

SoluSoft helps government agencies tackle fraud faster

Fraud, in all its forms, is a pervasive problem, spanning industries and preying on vulnerabilities in federal and state government systems. Each year in the…

3 minute read

Stay in the loop!

Get our most popular content delivered monthly to your inbox.