Node Manager

Overview of Node Manager

The Managed Servers in a production Weblogic Server environment are often distributed across multiple machines and geographic locations.

Node Manager is a Java utility that runs as separate process from Weblogic Server and allows you to perform common operations tasks for a Managed Server, regardless of its location with respect to its Administration Server. While use of Node Manager is optional, it provides valuable benefits if your Weblogic Server environment hosts applications with high availability requirements.

If you run Node Manager on a machine that hosts Managed Servers, you can start and stop the Managed Servers remotely using the Administration Console or from the command line. Node Manager can also automatically restart a Managed Server after an unexpected failure.

Accessing Node Manager

A Node Manager client can be local or remote to the Node Managers with which it communicates. You access either version of Node Manager—the Java version or the script-based (SSH) version—from the following clients. (In addition, an SSH client in the form of a shell command template is provided for use with the script-based Node Manager.)

• Administration Server
  • Administration Console, from the Environments>Machines>Configuration>Node Manager page.
  • JMX utilities you write yourself.

For more information about JMX, see Developing Custom Management Utilities with JMX.

• WLST commands and scripts—WLST offline serves as a Node Manager command-line interface that can run in the absence of a running Administration Server. You can use WLST commands to start, stop, and monitor a server instance without connecting to an Administration Server. Starting the Administration Server is the main purpose of the stand-alone client. However, you can also use it to:
  • Stop a server instance that was started by Node Manager.
  • Start a Managed Server.
  • Access the contents of a Node Manager log file.
  • Obtain server status.
  • Retrieve the contents of server output log

What You Can Do with Node Manager

Start, Shut Down, and Restart an Administration Server

Using the Weblogic Scripting Tool (or SSH client for Script-based Node Manager only), you connect to the Node Manager process on the machine that hosts the Administration Server and issue commands to start, shut down, or restart an Administrative Server. The relationship of an Administration Server to Node Manager varies for different scenarios.

• An Administration Server can be under Node Manager control—You can start it, monitor it, and restart it using Node Manager.
• An Administration Server can be a Node Manager client—When you start or stop Managed Servers from the Administration Console, you are accessing Node Manager via the Administration Server.
• An Administration Server supports the process of starting up a Managed Server with Node Manager—When you start a Managed Server with Node Manager, the Managed Server contacts the Administration Server to obtain outstanding configuration updates.

Start, Shut Down, Suspend, and Restart Managed Servers

From the Weblogic Server Scripting Tool (WLST) command line or scripts, you can issue commands to Node Manager to start, shut down, suspend, and restart Managed Server instances and clusters.

Node Manager can restart a Managed Server after failure even when the Administration Server is unavailable if Managed Server Independence (MSI) mode is enabled for that Managed Server instance. This is enabled by default.

Restart Administration and Managed Servers

If a server instance that was started using Node Manager fails, Node Manager automatically restarts it.

Note: Node Manager can only restart a server that was started via Node Manager.

The restart feature is configurable. Node Manager's default behavior is to:

• Automatically restart server instances under its control that fail. You can disable this feature.
• Restart failed server instances no more than a specific number of times. You define the number of restarts by setting the RestartMax property in the Node Managerstartup.properties file.

If Node Manager fails or is explicitly shut down, upon restart, it determines the server instances that were under its control when it exited. Node Manager can restart any failed server instances as necessary.

Note: It is advisable to run Node Manager as an operating system service, so that it restarts automatically if its host machine is restarted.

Monitor Servers and View Log Data

Node Manager creates a log file for the Node Manager process and a log file of server output for each server instance it controls. You can view these log files, as well as log files for a server instance using the Administration Console or WLST commands.

Diagram of Node Manager and Servers

How Node Manager Starts an Administration Server

Node Manager is running on Machine A, which hosts the Administration Server. The stand-alone Node Manager client is remote.

1. An authorized user issues the WLST offline command, nmConnect to connect to the Node Manager process on the machine that hosts the Administration Server, and issues a command to start the Administration Server. (If the Node Manager instance is the SSH version, the user can connect using the SSH client).

The start command identifies the domain and server instance to start, and in the case of the Java Node Manager, provides the Node Manager username and password.

Note: If the user has previously connected to the Node Manager, a boot.properties file exists, and the user does not have to supply username and password.

2. Node Manager looks up the domain directory in nodemanager.domains, and authenticates the user credentials using a local file that contains the encrypted username and password.

3. Node Manager creates the Administration Server process.

4. The Administration Server obtains the domain configuration from its config directory.

How Node Manager Starts a Managed Server

Node Manager is running on Machine B, which hosts Managed Server 1. The Administration Server for the domain is running on Machine A.

1. From the Administration Console, the user issues a start command for Managed Server 1.

Note: A stand-alone client can also issue a start command for a Managed Server.

2. The Administration Server issues a start command for Managed Server 1 to the Node Manager on the Machine B, providing the remote start properties configured for Managed Server 1.

3. Node Manager starts Managed Server 1.

Node Manager starts the Managed Server using the same root directory where the Node Manager process is running. To run the Managed Server in a different directory, set the Root Directory attribute in the Server—>Configuration—>Server Start console page.

4. Managed Server 1 contacts the Administration Server to check for updates to its configuration information.

5. If there are outstanding changes to the domain configuration, Managed Server 1 updates its local cache of configuration data.

How Node Manager Restarts an Administration Server

Node Manager is running on the machine that hosts the Administration Server. The Administration Server, which was initially started with Node Manager, has exited. The Administration Server's AutoRestart attribute is set to true.

Note: If a server instance's AutoRestart attribute is set to false, Node Manager will not restart it.

1. Node Manager determines from the Administration Server process exit code that it requires restart.

2. Node Manager obtains the username and password for starting the Administration Server from the boot.properties file, and the server startup properties from the<server_name>/data/nodemanager/startup.properties file.

3. Node Manager starts the Administration Server.

4. The Administration Server reads its configuration data and starts up.

How Node Manager Restarts a Managed Server

Node Manager is running on Machine B, which hosts Managed Server 1. Managed Server 1, which was initially started with Node Manager, has exited. Managed Server 1's AutoRestartattribute is set to true.

Note: If a server instance's AutoRestart attribute is set to false, Node Manager will not restart it.

1. Node Manager determines from Managed Server 1's last known state that it requires restarting.

2. Node Manager obtains the username and password for starting Managed Server 1 from the boot.properties file, and the server startup properties from thestartup.properties file. These server-specific files are located in the server directory for Managed Server 1.

3. Node Manager starts Managed Server 1.

Note: Node Manager waits RestartDelaySeconds after a server instances fails before attempting to restart it.

4. Managed Server 1 attempts to contact the Administration Server to check for updates to its configuration data. If it contacts the Administration Server and obtains updated configuration data, it updates its local cache of the config directory.

5. If Managed Server 1 fails to contact the Administration Server, and if Managed Server Independence mode (MSI) is enabled, Managed Server 1 uses its locally cached configuration data.

Note: Managed Server Independence mode is enabled by default.

Node Manager-Defined States for Restarting Managed Servers

Node Manager defines its own, internal Managed Server states for use when restarting a server. If Node Manager is configured to restart Managed Servers, you may observe these states in the Administration Console during the restart process.

• FAILED_RESTARTING—Indicates that Node Manager is currently restarting a failed Managed Server.
• FAILED_NOT_RESTARTABLE—Indicates that the Managed Server has failed or was killed by Node Manager as a result of the Managed Server's AutoKillIfFailed attribute being set to True, but Node Manager cannot restart the Managed Server because its AutoRestart attribute is set to False.

This state may also occur if a server fails during startup.

How Node Manager Shuts Down a Server Instance

Node Manager is running on Machine B, which hosts Managed Server 1.

1. Through the Administration Console, an authorized user issues a shutdown command for Managed Server 1.

2. The Administration Server issues the shutdown command directly to Managed Server 1. If it successfully contacts Managed Server 1, Managed Server 1 performs the shutdown sequence described in Graceful Shutdown in Managing Server Startup and Shutdown.

3. If, in the previous step, the Administration Server failed to contact Managed Server 1, it issues a shutdown command for Managed Server 1 to Node Manager on Machine B.

4. Node Manager issues a request to the operating system to kill Managed Server 1.

5. The operating system ends the Managed Server 1 process.

Node Manager and System Crash Recovery

The CrashRecoveryEnabled configuration property allows Node Manager to recover from a system crash. The property is enable by default.

After the system is restarted, Node Manager checks each managed domain specified in the nodemanager.domains file to determine if there are any server instances that were not cleanly shutdown. This is determined by the presence of any lock files which are created by Node Manager when a Weblogic Server process is created. This lock file contains the process identifier for Weblogic Server startup script. If the lock file exists, but the process ID is not running, Node Manager will attempt to automatically restart the server.

If the process is running, Node Manager performs an additional check to access the management servlet running in the process to verify that the process corresponding to the process ID is a Weblogic Server instance.

Note: When Node Manager performs a check to access the management servlet, an alert may appear in the server log regarding improper credentials.

Diagram of Node Manager Configuration and Log Files

In managing multiple servers, Node Manager uses multiple configuration files and outputs log files to multiple directories, as shown in the following figure.