2.11 Tracing

The tracing facilities defined in IEEE Std 1003.1-2001 allow a process to select a set of trace event types, to activate a trace stream of the selected trace events as they occur in the flow of execution, and to retrieve the recorded trace events.

The tracing operation relies on three logically different components: the traced process, the controller process, and the analyzer process. During the execution of the traced process, when a trace point is reached, a trace event is recorded into the trace streams created for that process in which the associated trace event type identifier is not being filtered out. The controller process controls the operation of recording the trace events into the trace stream. It shall be able to:

• Initialize the attributes of a trace stream

• Create the trace stream (for a specified traced process) using those attributes

• Start and stop tracing for the trace stream

• Filter the type of trace events to be recorded, if the Trace Event Filter option is supported

• Shut a trace stream down

These operations can be done for an active trace stream. The analyzer process retrieves the traced events either at runtime, when the trace stream has not yet been shut down, but is still recording trace events; or after opening a trace log that had been previously recorded and shut down. These three logically different operations can be performed by the same process, or can be distributed into different processes.

A trace stream identifier can be created by a call to posix_trace_create(), posix_trace_create_withlog(), or posix_trace_open(). The posix_trace_create() and posix_trace_create_withlog() functions should be used by a controller process. The posix_trace_open() should be used by an analyzer process.

The tracing functions can serve different purposes. One purpose is debugging the possibly pre-instrumented code, while another is post-mortem fault analysis. These two potential uses differ in that the first requires pre-filtering capabilities to avoid overwhelming the trace stream and permits focusing on expected information; while the second needs comprehensive trace capabilities in order to be able to record all types of information.

The events to be traced belong to two classes:

1. User trace events (generated by the application instrumentation)

2. System trace events (generated by the operating system)

The trace interface defines several system trace event types associated with control of and operation of the trace stream. This small set of system trace events includes the minimum required to interpret correctly the trace event information present in the stream. Other desirable system trace events for some particular application profile may be implemented and are encouraged; for example, process and thread scheduling, signal occurrence, and so on.

Each traced process shall have a mapping of the trace event names to trace event type identifiers that have been defined for that process. Each active trace stream shall have a mapping that incorporates all the trace event type identifiers predefined by the trace system plus all the mappings of trace event names to trace event type identifiers of the processes that are being traced into that trace stream. These mappings are defined from the instrumented application by calling the posix_trace_eventid_open() function and from the controller process by calling the posix_trace_trid_eventid_open() function. For a pre-recorded trace stream, the list of trace event types is obtained from the pre-recorded trace log.

The st_ctime and st_mtime fields of a file associated with an active trace stream shall be marked for update every time any of the tracing operations modifies that file.

The st_atime field of a file associated with a trace stream shall be marked for update every time any of the tracing operations causes data to be read from that file.

Results are undefined if the application performs any operation on a file descriptor associated with an active or pre-recorded trace stream until posix_trace_shutdown() or posix_trace_close() is called for that trace stream.

The main purpose of this option is to define a complete set of functions and concepts that allow a conforming application to be traced from creation to termination, whatever its realtime constraints and properties.

2.11.1 Tracing Data Definitions

Structures

The <trace.h> header shall define the posix_trace_status_info and posix_trace_event_info structures described below. Implementations may add extensions to these structures.

posix_trace_status_info Structure

To facilitate control of a trace stream, information about the current state of an active trace stream can be obtained dynamically. This structure is returned by a call to the posix_trace_get_status() function.

The posix_trace_status_info structure defined in <trace.h> shall contain at least the following members:

If the Trace Log option is supported in addition to the Trace option, the posix_trace_status_info structure defined in <trace.h> shall contain at least the following additional members:

The posix_stream_status member indicates the operating mode of the trace stream and shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_RUNNING

Tracing is in progress; that is, the trace stream is accepting trace events.

POSIX_TRACE_SUSPENDED

The trace stream is not accepting trace events. The tracing operation has not yet started or has stopped, either following a posix_trace_stop() function call or because the trace resources are exhausted.

The posix_stream_full_status member indicates the full status of the trace stream, and it shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_FULL

The space in the trace stream for trace events is exhausted.

POSIX_TRACE_NOT_FULL

There is still space available in the trace stream.

The combination of the posix_stream_status and posix_stream_full_status members also indicates the actual status of the stream. The status shall be interpreted as follows:

POSIX_TRACE_RUNNING and POSIX_TRACE_NOT_FULL

This status combination indicates that tracing is in progress, and there is space available for recording more trace events.

POSIX_TRACE_RUNNING and POSIX_TRACE_FULL

This status combination indicates that tracing is in progress and that the trace stream is full of trace events. This status combination cannot occur unless the stream-full-policy is set to POSIX_TRACE_LOOP. The trace stream contains trace events recorded during a moving time window of prior trace events, and some older trace events may have been overwritten and thus lost.

POSIX_TRACE_SUSPENDED and POSIX_TRACE_NOT_FULL

This status combination indicates that tracing has not yet been started, has been stopped by the posix_trace_stop() function, or has been cleared by the posix_trace_clear() function.

POSIX_TRACE_SUSPENDED and POSIX_TRACE_FULL

This status combination indicates that tracing has been stopped by the implementation because the stream-full-policy attribute was POSIX_TRACE_UNTIL_FULL and trace resources were exhausted, or that the trace stream was stopped by the function posix_trace_stop() at a time when trace resources were exhausted.

The posix_stream_overrun_status member indicates whether trace events were lost in the trace stream, and shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_OVERRUN

At least one trace event was lost and thus was not recorded in the trace stream.

POSIX_TRACE_NO_OVERRUN

No trace events were lost.

When the corresponding trace stream is created, the posix_stream_overrun_status member shall be set to POSIX_TRACE_NO_OVERRUN.

Whenever an overrun occurs, the posix_stream_overrun_status member shall be set to POSIX_TRACE_OVERRUN.

An overrun occurs when:

• The policy is POSIX_TRACE_LOOP and a recorded trace event is overwritten.

• The policy is POSIX_TRACE_UNTIL_FULL and the trace stream is full when a trace event is generated.

• If the Trace Log option is supported, the policy is POSIX_TRACE_FLUSH and at least one trace event is lost while flushing the trace stream to the trace log.

The posix_stream_overrun_status member is reset to zero after its value is read.

If the Trace Log option is supported in addition to the Trace option, the posix_stream_flush_status, posix_stream_flush_error, posix_log_overrun_status, and posix_log_full_status members are defined as follows; otherwise, they are undefined.

The posix_stream_flush_status member indicates whether a flush operation is being performed and shall have one of the following values defined by manifest constants in the header <trace.h>:

POSIX_TRACE_FLUSHING

The trace stream is currently being flushed to the trace log.

POSIX_TRACE_NOT_FLUSHING

No flush operation is in progress.

The posix_stream_flush_status member shall be set to POSIX_TRACE_FLUSHING if a flush operation is in progress either due to a call to the posix_trace_flush() function (explicit or caused by a trace stream shutdown operation) or because the trace stream has become full with the stream-full-policy attribute set to POSIX_TRACE_FLUSH. The posix_stream_flush_status member shall be set to POSIX_TRACE_NOT_FLUSHING if no flush operation is in progress.

The posix_stream_flush_error member shall be set to zero if no error occurred during flushing. If an error occurred during a previous flushing operation, the posix_stream_flush_error member shall be set to the value of the first error that occurred. If more than one error occurs while flushing, error values after the first shall be discarded. The posix_stream_flush_error member is reset to zero after its value is read.

The posix_log_overrun_status member indicates whether trace events were lost in the trace log, and shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_OVERRUN

At least one trace event was lost.

POSIX_TRACE_NO_OVERRUN

No trace events were lost.

When the corresponding trace stream is created, the posix_log_overrun_status member shall be set to POSIX_TRACE_NO_OVERRUN. Whenever an overrun occurs, this status shall be set to POSIX_TRACE_OVERRUN. The posix_log_overrun_status member is reset to zero after its value is read.

The posix_log_full_status member indicates the full status of the trace log, and it shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_FULL

The space in the trace log is exhausted.

POSIX_TRACE_NOT_FULL

There is still space available in the trace log.

The posix_log_full_status member is only meaningful if the log-full-policy attribute is either POSIX_TRACE_UNTIL_FULL or POSIX_TRACE_LOOP.

For an active trace stream without log, that is created by the posix_trace_create() function, the posix_log_overrun_status member shall be set to POSIX_TRACE_NO_OVERRUN and the posix_log_full_status member shall be set to POSIX_TRACE_NOT_FULL.

posix_trace_event_info Structure

The trace event structure posix_trace_event_info contains the information for one recorded trace event. This structure is returned by the set of functions posix_trace_getnext_event(), posix_trace_timedgetnext_event(), and posix_trace_trygetnext_event().

The posix_trace_event_info structure defined in <trace.h> shall contain at least the following members:

In addition, if the Trace option and the Threads option are both supported, the posix_trace_event_info structure defined in <trace.h> shall contain the following additional member:

The posix_event_id member represents the identification of the trace event type and its value is not directly defined by the user. This identification is returned by a call to one of the following functions: posix_trace_trid_eventid_open(), posix_trace_eventtypelist_getnext_id(), or posix_trace_eventid_open(). The name of the trace event type can be obtained by calling posix_trace_eventid_get_name().

The posix_pid is the process identifier of the traced process which generated the trace event. If the posix_event_id member is one of the implementation-defined system trace events and that trace event is not associated with any process, the posix_pid member shall be set to zero.

For a user trace event, the posix_prog_address member is the process mapped address of the point at which the associated call to the posix_trace_event() function was made. For a system trace event, if the trace event is caused by a system service explicitly called by the application, the posix_prog_address member shall be the address of the process at the point where the call to that system service was made.

The posix_truncation_status member defines whether the data associated with a trace event has been truncated at the time the trace event was generated, or at the time the trace event was read from the trace stream, or (if the Trace Log option is supported) from the trace log (see the event argument from the posix_trace_getnext_event() function). The posix_truncation_status member shall have one of the following values defined by manifest constants in the <trace.h> header:

POSIX_TRACE_NOT_TRUNCATED

All the traced data is available.

POSIX_TRACE_TRUNCATED_RECORD

Data was truncated at the time the trace event was generated.

POSIX_TRACE_TRUNCATED_READ

Data was truncated at the time the trace event was read from a trace stream or a trace log because the reader's buffer was too small. This truncation status overrides the POSIX_TRACE_TRUNCATED_RECORD status.

The posix_timestamp member shall be the time at which the trace event was generated. The clock used is implementation-defined, but the resolution of this clock can be retrieved by a call to the posix_trace_attr_getclockres() function.

If the Threads option is supported in addition to the Trace option:

• The posix_thread_id member is the identifier of the thread that generated the trace event. If the posix_event_id member is one of the implementation-defined system trace events and that trace event is not associated with any thread, the posix_thread_id member shall be set to zero.

Otherwise, this member is undefined.

Trace Stream Attributes

Trace streams have attributes that compose the posix_trace_attr_t trace stream attributes object. This object shall contain at least the following attributes:

• The generation-version attribute identifies the origin and version of the trace system.

• The trace-name attribute is a character string defined by the trace controller, and that identifies the trace stream.

• The creation-time attribute represents the time of the creation of the trace stream.

• The clock-resolution attribute defines the clock resolution of the clock used to generate timestamps.

• The stream-min-size attribute defines the minimum size in bytes of the trace stream strictly reserved for the trace events.

• The stream-full-policy attribute defines the policy followed when the trace stream is full; its value is POSIX_TRACE_LOOP, POSIX_TRACE_UNTIL_FULL, or POSIX_TRACE_FLUSH.

• The max-data-size attribute defines the maximum record size in bytes of a trace event.

In addition, if the Trace option and the Trace Inherit option are both supported, the posix_trace_attr_t trace stream creation attributes object shall contain at least the following attributes:

• The inheritance attribute specifies whether a newly created trace stream will inherit tracing in its parent's process trace stream. It is either POSIX_TRACE_INHERITED or POSIX_TRACE_CLOSE_FOR_CHILD.

In addition, if the Trace option and the Trace Log option are both supported, the posix_trace_attr_t trace stream creation attributes object shall contain at least the following attribute:

• If the file type corresponding to the trace log supports the POSIX_TRACE_LOOP or the POSIX_TRACE_UNTIL_FULL policies, the log-max-size attribute defines the maximum size in bytes of the trace log associated with an active trace stream. Other stream data-for example, trace attribute values-shall not be included in this size.

• The log-full-policy attribute defines the policy of a trace log associated with an active trace stream to be POSIX_TRACE_LOOP, POSIX_TRACE_UNTIL_FULL, or POSIX_TRACE_APPEND.

2.11.2 Trace Event Type Definitions

System Trace Event Type Definitions

The following system trace event types, defined in the <trace.h> header, track the invocation of the trace operations:

• POSIX_TRACE_START shall be associated with a trace start operation.

• POSIX_TRACE_STOP shall be associated with a trace stop operation.

• If the Trace Event Filter option is supported, POSIX_TRACE_FILTER shall be associated with a trace event type filter change operation.

The following system trace event types, defined in the <trace.h> header, report operational trace events:

• POSIX_TRACE_OVERFLOW shall mark the beginning of a trace overflow condition.

• POSIX_TRACE_RESUME shall mark the end of a trace overflow condition.

• If the Trace Log option is supported, POSIX_TRACE_FLUSH_START shall mark the beginning of a flush operation.

• If the Trace Log option is supported, POSIX_TRACE_FLUSH_STOP shall mark the end of a flush operation.

• If an implementation-defined trace error condition is reported, it shall be marked POSIX_TRACE_ERROR.

The interpretation of a trace stream or a trace log by a trace analyzer process relies on the information recorded for each trace event, and also on system trace events that indicate the invocation of trace control operations and trace system operational trace events.

The POSIX_TRACE_START and POSIX_TRACE_STOP trace events specify the time windows during which the trace stream is running.

• The POSIX_TRACE_STOP trace event with an associated data that is equal to zero indicates a call of the function posix_trace_stop().

• The POSIX_TRACE_STOP trace event with an associated data that is different from zero indicates an automatic stop of the trace stream (see posix_trace_attr_getstreamfullpolicy() ).

The POSIX_TRACE_FILTER trace event indicates that a trace event type filter value changed while the trace stream was running.

The POSIX_TRACE_ERROR serves to inform the analyzer process that an implementation-defined internal error of the trace system occurred.

The POSIX_TRACE_OVERFLOW trace event shall be reported with a timestamp equal to the timestamp of the first trace event overwritten. This is an indication that some generated trace events have been lost.

The POSIX_TRACE_RESUME trace event shall be reported with a timestamp equal to the timestamp of the first valid trace event reported after the overflow condition ends and shall be reported before this first valid trace event. This is an indication that the trace system is reliably recording trace events after an overflow condition.

Each of these trace event types shall be defined by a constant trace event name and a trace_event_id_t constant; trace event data is associated with some of these trace events.

If the Trace option is supported and the Trace Event Filter option and the Trace Log option are not supported, the following predefined system trace events in Trace Option: System Trace Events shall be defined:

If the Trace option and the Trace Event Filter option are both supported, and if the Trace Log option is not supported, the following predefined system trace events in Trace and Trace Event Filter Options: System Trace Events shall be defined:

If the Trace option and the Trace Log option are both supported, and if the Trace Event Filter option is not supported, the following predefined system trace events in Trace and Trace Log Options: System Trace Events shall be defined:

If the Trace option, the Trace Event Filter option, and the Trace Log option are all supported, the following predefined system trace events in Trace, Trace Log, and Trace Event Filter Options: System Trace Events shall be defined:

User Trace Event Type Definitions

The user trace event POSIX_TRACE_UNNAMED_USEREVENT is defined in the <trace.h> header. If the limit of per-process user trace event names represented by {TRACE_USER_EVENT_MAX} has already been reached, this predefined user event shall be returned when the application tries to register more events than allowed. The data associated with this trace event is application-defined.

The following predefined user trace event in Trace Option: User Trace Event shall be defined:

2.11.3 Trace Functions

The trace interface is built and structured to improve portability through use of trace data of opaque type. The object-oriented approach for the manipulation of trace attributes and trace event type identifiers requires definition of many constructor and selector functions which operate on these opaque types. Also, the trace interface must support several different tracing roles. To facilitate reading the trace interface, the trace functions are grouped into small functional sets supporting the three different roles:

• A trace controller process requires functions to set up and customize all the resources needed to run a trace stream, including:

  • Attribute initialization and destruction ( posix_trace_attr_init())

  • Identification information manipulation ( posix_trace_attr_getgenversion())

  • Trace system behavior modification ( posix_trace_attr_getinherited())

  • Trace stream and trace log size set ( posix_trace_attr_getmaxusereventsize())

  • Trace stream creation, flush, and shutdown ( posix_trace_create())

  • Trace stream and trace log clear ( posix_trace_clear())

  • Trace event type identifier manipulation ( posix_trace_trid_eventid_open())

  • Trace event type identifier list exploration ( posix_trace_eventtypelist_getnext_id())

  • Trace event type set manipulation ( posix_trace_eventset_empty())

  • Trace event type filter set ( posix_trace_set_filter())

  • Trace stream start and stop ( posix_trace_start())

  • Trace stream information and status read ( posix_trace_get_attr())

• A traced process requires functions to instrument trace points:

  • Trace event type identifiers definition and trace points insertion ( posix_trace_event())

• A trace analyzer process requires functions to retrieve information from a trace stream and trace log:

  • Identification information read ( posix_trace_attr_getgenversion())

  • Trace system behavior information read ( posix_trace_attr_getinherited())

  • Trace stream and trace log size get ( posix_trace_attr_getmaxusereventsize())

  • Trace event type identifier manipulation ( posix_trace_trid_eventid_open())

  • Trace event type identifier list exploration ( posix_trace_eventtypelist_getnext_id())

  • Trace log open, rewind, and close ( posix_trace_open())

  • Trace stream information and status read ( posix_trace_get_attr())

  • Trace event read ( posix_trace_getnext_event())