Analytics

What Should I do When the Database Node is Down?

When database node is DOWN, troubleshoot using the following checklist. Step Task Results 1 Check whether your database is UP. $ admintools -t db_status -s…

OpenText

June 14, 2018•2 minute read

Shot of two businesswomen using a digital tablet together during a collaboration at work

When database node is DOWN, troubleshoot using the following checklist.

Step	Task	Results
1	Check whether your database is UP. `$ admintools -t db_status -s UP`	If the database is UP, go to Step 2. If the database if not UP, restart your database. `$ admintools -t start_db -d <Database_name> -p <Database_password>` If the database starts, the checklist is complete. If the database does not start, see the Database Process Not Starting checklist.
2	Identify all the DOWN nodes. `=> SELECT node_name, node_address, node_state FROM nodes WHERE node_state = 'DOWN';`	Upon identification of all the DOWN node/s, proceed to Step 3.
3	Check whether you can establish a connection with the DOWN nodes using SSH. `$ ssh dbadmin @<nodedown_ip>`	If you can SSH into the node, restart Vertica process on the DOWN node. `$ admintools -t restart_node -d <database_name> -s <node_host_name or IP>` If the restart was successful, the checklist is complete. If the restart failed, go to Step 4. If you cannot SSH into the node, contact your system administrator to find if it is a port issue or a network issue.
4	To find reasons for restart failure, on the DOWN node tail startup.log. A snippet of the tail startup.log is as follows: `$ tail -f catalog-path/database-name/v_database-name_node_catalog/startup.log { "node" : "v_cdmt0_node0001", "stage" : "Database Halted", "text" : Data consistency problems found; Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data. "timestamp" : "2016-07-31 18:17:04.122" }`	The log results shows the latest state of the DOWN node. Proceed to Step 5 to see these stages.
5	If the startup.log… a. Remains in the Waiting for cluster invitestage.	See the Spread Debugging Checklist.
	b. Remains in the Recovery stage.	See the Node Recovery Checklist.
	c. Shows an error message, Data inconsistency problems found, restart the node with the force option. `$ admintools -t restart_node -d Database_name -s node_name --force`	Upon restart the checklist is complete.
	d. Shows no new data in the startup.log after last restart, check dbLog file for errors.	If you cannot resolve errors, contact Vertica Support.
	e. Shows Shutdown Complete but the node is still DOWN. tail vertica.log and look for <ERROR> and <PANIC>.	Contact Vertica Support with PANIC report, ErrorReport.txt, and scrutinize.

Share this post

OpenText

OpenText, The Information Company, enables organizations to gain insight through market-leading information management solutions, powered by OpenText Cloud Editions.

See all posts