File Locking over XNFS

Introduction

Because NFS is a stateless service, it cannot provide inherently stateful services such as file locking and access control synchronisation. Instead these services are provided by two cooperating processes: the Network Lock Manager (NLM) and the Network Status Monitor (NSM). The NLM and NSM are RPC-based servers which normally execute as autonomous "daemon" servers on XNFS client and server systems. They work together to provide stateful file locking and access control capability over XNFS. This chapter describes the RPC protocols which the NLM and NSM implement, and defines how they interact. Full specifications of the NLM and NSM protocols are in Network Lock Manager Protocol and Network Status Monitor Protocol .

NLM Protocol

The NLM is a service that provides advisory X/Open CAE file and record locking, and DOS compatible file sharing and locking in an XNFS environment. Its use is strongly encouraged but not mandatory. XNFS clients must be prepared to interoperate with servers which do not support this service. It is also recommended, but not required, that locks created by DOS processes are honoured by processes running on an X/Open host and vice versa.

The NLM provides two types of locks, monitored and non-monitored.

Monitored Locks

Monitored locks are reliable. A client process which establishes monitored locks can be assured that if the server host, on which the locks are established, crashes and recovers, the locks will be reinstated without any action on the client process' part. Likewise, locks that are held by a client process will be discarded by the NLM on the server host if the client host crashes before the locks are released.

Monitored locks require both the client and server hosts to implement the NSM protocol.

Monitored locks are preferred over the non-monitored locks.

Non-monitored Locks

Non-monitored locks are provided to support single-tasking personal computers that cannot run an NSM due to memory or speed constraints. A client that is able to run an NSM should use monitored locks.

Non-monitored locks provide the same functionality as monitored locks except if the server host, on which the locks are established, crashes and recovers, the locks will not be re-established. The personal computer client is responsible for detecting a server host failure and re-establishing the locks. Additionally, the personal computer client must inform the server NLM when it has been rebooted so it can discard all locks and file shares held for the client.

NSM Protocol

The NSM is a service that provides applications with information on the status of network hosts. It is included in this document as it is heavily used by the NLM to track hosts that have established locks and the hosts that are holding those locks. Although the NSM is a general service, this document will only describe the NSM as it is used by the NLM.

Each NSM keeps track of its own "state" and notifies any interested party of a change in this state. The state is merely a number which increases monotonically each time the condition of the host changes: an even number indicates the host is down, while an odd number indicates the host is up.

The NSM does not actively "probe" hosts it has been asked to monitor; instead it waits for the monitored host to notify it that the monitored host's status has changed (that is, crashed and rebooted).

When it receives an SM_MON request an NSM adds the information in the SM_MON parameter to a notify list. If the host has a status change (crashes and recovers), the NSM will notify each host on the notify list via the SM_NOTIFY call. If the NSM receives notification of a status change from another host it will search the notify list for that host and call the RPC supplied in the SM_MON call.

For obvious reasons, the NSM maintains copies of its current state and of the notify list on stable storage.

For correct operation of the NLM, the client and server hosts are required to monitor each other. When a lock request is issued by a process running on the client host, the NLM on the client host requests the NSM on the client host to monitor the server host. The client NLM then transmits the lock request to the NLM on the server. On reception of the lock request the NLM on the server host will request the NSM on the server host to monitor the client host. In this way each host is monitored by the NSM on the other host.

Interaction

It is assumed that the user process requests locks and file shares via a user-level API or system call such as the XSI fcntl(). The NLM protocol provides both synchronous and asynchronous procedures. An implementor may choose to use either the synchronous or asynchronous procedures to implement client functionality, but must be able to accept and process both types of requests.

This section will describe the interaction between the NLM and the NSM for the synchronous procedures. For simplicity it is assumed that all network lock requests are passed to the client NLM for handling. This is an implementation dependency; the protocol does not require this.

Monitored Locks

Monitored locks require both the client and server hosts to support the NSM protocol.

Locking

NLM_LOCK requests may be blocking or non-blocking. When the server NLM receives the NLM_LOCK request, it must make a call to the SM_MON procedure on its local NSM to monitor the calling host. The SM_MON call includes the name of the host to be monitored and an RPC to be called if the NSM is notified of a state change for the monitored host. The RPC information includes a transport end-point, program number, program version, procedure number and an opaque argument. The RPC information is of significance only to an NLM implementation, and is not defined by this specification.

If the lock can be granted immediately, or the call was non-blocking, the RPC returns immediately with the appropriate status (granted or denied).

If the lock cannot be granted immediately (it conflicts with an existing lock) and the call was a blocking call, the RPC will return with a blocked status, thus allowing the client NLM to continue processing. At this point the client NLM can choose to cancel the outstanding lock request by calling the NLM_CANCEL procedure. Upon reception of an NLM_CANCEL request, the server NLM will then delete the outstanding lock request and may request its local NSM to stop monitoring the calling host by calling the SM_UNMON procedure.

When the blocked lock request can be processed, the server NLM makes an NLM_GRANTED call-back to the client NLM indicating success or failure.

Once the lock has been granted, the client NLM instructs the local NSM to monitor the server via the SM_MON RPC, as described above; once again, the RPC used for notification is not defined by this specification. At this point the NSMs on both the client and server hosts are monitoring each other.

Crash Recovery

When the server host crashes and is restarted, its NSM will go through the notify list and will call the SM_NOTIFY procedure for each of these hosts to inform them of the state change. Each local NSM that receives this SM_NOTIFY call will search their notify list and make the corresponding RPC supplied in the previous SM_MON call, to the interested parties. One of the interested parties will be the client NLM protocol implementation which will have supplied an RPC which can go through the steps necessary to re-establish the lost locks during the server NLM server's grace period.

The grace period is an implementation-dependent time during which the NLM implementation will only accept requests to re-establish locks or shares that were in effect at the time of the crash. During this period any other lock or share requests will be returned with a status indicating that the NLM is in the grace period and is not accepting new requests.

If the client host crashes, upon reboot, the NSM will go through the same process notifying the NSMs on hosts in the notify list via the SM_NOTIFY procedure call that there was a change in state. The server NSM will receive this notification call and in turn notify the server NLM, via the provided RPC, that the client host had crashed. The server NLM can then dispose of all locks and shares held by the crashed host.

Unlocking

A monitored lock is unlocked by making a call to the NLM_UNLOCK procedure. The server NLM will process the request and release the lock and return status. The server NLM can then ask its local NSM to stop monitoring the calling host via the SM_UNMON procedure call. At this point the server NLM will check existing blocked lock requests and service them if possible.

Non-Monitored Locks

Non-monitored locks do not require the client or server host to support the NSM protocol. All non-monitored locks calls are synchronous.

Locking

A client host establishes a non-monitored lock by calling the NLM_NM_LOCK procedure on the server NLM. The server NLM will process the lock and return status to the client host indicating whether the lock was granted or denied. The NLM_NM_LOCK procedure call cannot block and so cannot result in a call-back.

Crash Recovery

If the client host crashes while it has established locks or file shares, it must generate an NLM_FREE_ALL RPC upon reboot. When it receives an NLM_FREE_ALL request the server NLM must free all locks and files shares held by the requesting host. The client host has no way of determining whether the server host crashes, and therefore no way to re-establish the locks during the server NLM grace period. The client must be prepared to handle errors when a previously requested lock is lost.

Unlocking

The unlock operation is the same as for monitored locks. The NLM_UNLOCK procedure is called. The server NLM will process the request and release the lock as requested, and return a status.

Transport Issues

NLM and NSM implementations are required to support both UDP and TCP transports. Personal computer NFS clients will always use UDP when issuing locking and sharing requests to an NLM. Most implementations of the NSM use TCP when interacting with the local NLM and any remote NSMs, but both NLM and NSM must accept any request over either transport.

Examples of Locking

This section outlines the behaviour of the NLM and NSM daemons during an X/Open-compliant system boot, crash and reboot. It describes the following two cases:

• a client locks a file, and the server crashes and is restarted while the file is locked

• a client locks a file, then crashes and is restarted without releasing the lock.

Server Crash Example

Server NSM Initialisation

The server NSM is started. (Note that NSM and NLM initialisation proceed in parallel.) The server NSM retrieves a copy of the last server state from stable storage, increments it to the next odd value, and saves it on stable storage. It then processes the notify list on stable storage, which is initially empty.

Server NLM Initialisation

The server NLM is started. It issues an SM_UNMON_ALL to the server NSM, from which it obtains a copy of the server state. It then enters grace period recovery state, waits for the grace period, and then enters normal service state.

Client NSM Initialisation

The client NSM is started. (Note that NSM and NLM initialisation proceed in parallel.) The client NSM retrieves a copy of the last client state from stable storage, increments it to the next odd value, and saves it on stable storage. It then processes the notify list, which is initially empty.

Client NLM Initialisation

The client NLM is started. It issues an SM_UNMON_ALL to the client NSM, from which it obtains a copy of the client state. It then enters grace period recovery state, waits for the grace period, and then enters normal service state.

Client Lock Request

A process on the client system requests a byte range lock on a file stored on the server. The client NFS passes this request, including the file handle, to the local NLM using a private protocol.

The client NLM issues an NLM_LOCK RPC to the server NLM. This request includes a copy of the client state.

The server NLM determines that the lock can be granted. It verifies that this is the first lock held for the client, and issues an SM_MON call to the server NSM instructing it to monitor the client.

The server NSM saves the client name and RPC information in the notify list, committing the client name to stable storage, and reports success to the server NLM.

The server NLM records the fact that it is holding a lock for the named client which is in client state. It then sends a success response to the client NLM.

The client NLM verifies that this is the first lock which it is holding on the server system, and issues an SM_MON call to the client NSM instructing it to monitor the server.

The client NSM saves the server name and RPC information in the notify list, committing the server name to stable storage, and reports success to the client NLM.

The client NLM records the fact that the server NLM is holding a lock for it, and returns to the local application.

Server Failure and Restart

The server system fails and is restarted.

Server NSM Restart

The server NSM is restarted. (Note that NSM and NLM initialisation proceed in parallel.) The server NSM retrieves a copy of the last server state from stable storage, increments it to the next odd value, and saves it on stable storage. It then processes the notify list on stable storage, adding each name to a recovery list.

For each name in the recovery list, the server NSM issues an SM_NOTIFY RPC to the NSM on the named host. In this example it will issue an SM_NOTIFY to the client NSM, including the server name and the new server state.

The client NSM receives the SM_NOTIFY RPC and compares the hostname against each entry in the notify list. When it encounters a match, it calls back the client NLM using the RPC information provided with the original SM_MON RPC.

The callback procedure in the client NLM notes that the server state has changed and schedules lock recovery (see below). It then acknowledges the RPC callback from the client NSM.

After comparing the name against all entries in the notify list, the client NSM acknowledges the SM_NOTIFY RPC from the server NSM.

After processing all entries in the recovery list, the server NSM enters normal service state with an empty notify list.

Server NLM Restart

The server NLM is started. It issues an SM_UNMON_ALL to the server NSM, from which it obtains a copy of the new server state. It then enters grace period recovery state.

Client Lock Recovery

The client NLM now attempts to recover all locks which it was holding on the server. For each lock, it issues an NLM_LOCK request with reclaim set to true. This NLM_LOCK is processed as described above; the server NLM and NSM register the lock and (re)initiate monitoring of the client, and the client NLM confirms the lock and arranges for the client NSM to monitor the server.

Server NLM Restart Completion

At the conclusion of the grace period, the server NLM enters normal service mode.

Client Unlock Request

Eventually the client application releases the lock on the file. The client NLM issues an NLM_UNLOCK RPC to the server NLM.

The server NLM releases the lock, and notices that this is the last lock which was being held on behalf of the client. It issues an SM_UNMON RPC to the server NSM.

The server NSM removes the client from the notify list, and returns to the server NLM, which completes the NLM_UNLOCK request.

The client NLM deletes the lock record, and notices that this is the last lock which it was holding on the server. It issues an SM_UNMON RPC to the client NSM.

The client NSM removes the server from the notify list, and returns to the client NLM, which completes the application's unlock request.

Client Crash Example

Initialisation

The server and client NSM and NLM are initialised as described in the previous example.

Client Lock Request

The client lock request is processed as described in the previous example.

Client Failure and Restart

The client system fails and is restarted.

Client NSM Restart

The client NSM is restarted. (Note that NSM and NLM initialisation proceed in parallel.) The client NSM retrieves a copy of the last client state from stable storage, increments it to the next odd value, and saves it on stable storage. It then processes the notify list on stable storage, adding each name to a recovery list.

For each name in the recovery list, the client NSM issues an SM_NOTIFY RPC to the NSM on the named host. In this example it will issue an SM_NOTIFY to the server NSM, including the client name and the new client state.

The server NSM receives the SM_NOTIFY RPC and compares the hostname against each entry in the notify list. When it encounters a match, it calls back the server NLM using the RPC information provided with the original SM_MON RPC.

The callback procedure in the server NLM notes that the client state has changed and releases all locks held on behalf of the client. It then acknowledges the RPC callback from the server NSM.

After comparing the name against all entries in the notify list, the server NSM acknowledges the SM_NOTIFY RPC from the client NSM.

After processing all entries in the recovery list, the client NSM enters normal service state with an empty notify list.

Client NLM Restart

The client NLM is now restarted as normal.