When database node is DOWN, troubleshoot using the following checklist.
| Step | Task | Results |
|---|---|---|
| 1 | Check whether your database is UP.$ admintools -t db_status -s UP | If the database is UP, go to Step 2. If the database if not UP, restart your database. If the database starts, the checklist is complete. If the database does not start, see the Database Process Not Starting checklist. |
| 2 | Identify all the DOWN nodes.=> SELECT node_name, node_address, node_state FROM nodes WHERE node_state = 'DOWN'; | Upon identification of all the DOWN node/s, proceed to Step 3. |
| 3 | Check whether you can establish a connection with the DOWN nodes using SSH.$ ssh dbadmin @<nodedown_ip> | If you can SSH into the node, restart Vertica process on the DOWN node.$ admintools -t restart_node -d <database_name> -s <node_host_name or IP>
If you cannot SSH into the node, contact your system administrator to find if it is a port issue or a network issue. |
| 4 | To find reasons for restart failure, on the DOWN node tail startup.log. A snippet of the tail startup.log is as follows:$ tail -f catalog-path/database-name/v_database-name_node_catalog/startup.log | The log results shows the latest state of the DOWN node. Proceed to Step 5 to see these stages. |
| 5 | If the startup.log… a. Remains in the | See the Spread Debugging Checklist. |
| b. Remains in the Recovery stage. | See the Node Recovery Checklist. | |
c. Shows an error message, Data inconsistency problems found, restart the node with the force option.$ admintools -t restart_node -d Database_name -s node_name --force | Upon restart the checklist is complete. | |
| d. Shows no new data in the startup.log after last restart, check dbLog file for errors. | If you cannot resolve errors, contact Vertica Support. | |
| e. Shows Shutdown Complete but the node is still DOWN. tail vertica.log and look for <ERROR> and <PANIC>. | Contact Vertica Support with PANIC report, ErrorReport.txt, and scrutinize. |
