In a multi-threaded environment where data is shared between instances of a STREAMS component, synchronization of thread execution is necessary. WindNet STREAMS includes built-in facilities that free the development engineer from having to design into the STREAMS component synchronization services enabling access to shared resources. When discussing synchronization, both modules and drivers can be considered components. WindNet STREAMS synchronization relies on the following rules, as defined by the STREAMS programming specifications; refer to the UNIX System V Release 4 Programmer's Guide: STREAMS (Prentice Hall) and STREAMS Modules and Drivers (Prentice Hall).

• The STREAMS environment controls all component entry points completely. Component open, close, put, and service procedures can be called only from the STREAMS framework.

• The STREAMS definition explicitly states that components cannot assume a specific thread context in their put and service routines. This means any thread can execute a put or service routine, including threads from other user contexts or from other system threads.

• The STREAMS definition explicitly disallows components from invoking the WindNet STREAMS routine strmSleep( ) from put and service routines.

Based on these rules, the WindNet STREAMS synchronization model can provide the following services:

• Access to component instances (that is, STREAMS queues) is controlled by the STREAMS framework, and no longer requires you to develop code to achieve synchronization. In particular, most components written for a single-threaded SVR4 STREAMS environment can run in a multi-threaded environment, like WindNet STREAMS, without any code changes.

• In the typical case, multiple parallel threads of execution incur essentially no overhead, when the desired resource is available. Even if there is contention for a resource, requests are serialized and processed with minimal overhead.

• The design is optimized for efficient data movement: Put and service procedures incur low synchronization overhead. Access for open and close procedures is considered low frequency and demands higher overhead.

• Deadlocks cannot occur because execution threads never block. If a resource is unavailable, the execution thread which needs the resource simply adds itself to a list of jobs needing the resource. The thread which currently owns the resource will process this list of waiting jobs as part of its cleanup upon releasing control of the resource.

• A simple method is provided to ensure synchronized access to global data structures, such as routing tables, which are shared by different instances of a component or even separate components. For more information, §4.1.3.2 Updating Shared Data with strmSyncWriteAccess( ).

A STREAMS component written for an SVR3 environment assumes that only a single thread can execute at any one time. This type of component can operate in a multi-threaded environment like WindNet STREAMS. With minor modifications, some components written originally for single-threaded execution can take advantage of multi-threaded functionality for gains in performance. Both of these types of components require different levels of synchronization. Components written to handle multi-threaded execution take yet another synchronization level. WindNet STREAMS accommodates the extremes as well as the cases in between. Each STREAMS component ported to WindNet STREAMS must be installed and associated with a suitable synchronization level. Before assigning a synchronization level to a STREAMS component, make sure that you have read §4. Porting STREAMS Protocols to WindNet STREAMS.

Synchronization levels control the degree of parallelism allowed between separate threads of execution in a STREAMS component. The four constants listed in Table 5 specify levels of synchronization. Synchronization levels are assigned when configuring components into WindNet STREAMS; see §4.4 Configuration of STREAMS Drivers and Modules in WindNet STREAMS. Figure 13 shows two active instances of STREAMS Component A and Component B. Both instances form a STREAMS data path and execute under WindNet STREAMS. The explanations of the different levels of synchronization in the following subsections refer to the configuration of components in Figure 13. The synchronization level constants can be found in the file h/streams/sqlvl.h.

Table 5.   Synchronization Levels

Figure 13.   Component Instances and Queues

This is the least restrictive synchronization level. Separate threads of execution can run concurrently in the read-side and write-side queues of a component instance. In Figure 13, if Component A has SQLVL_QUEUE synchronization, four separate threads, one in each queue--A1, A2, A3, and A4--can be executing simultaneously; no two threads are permitted to execute simultaneously in any one queue of Component A.

Only one thread of execution is permitted in either the read-side or write-side queue of a component instance at a time. However, multiple instances may each have a separate concurrent thread of execution. In Figure 13, if Component A has SQLVL_QUEUEPAIR synchronization, two separate threads, one in queue A1 or A2, and another in A3 or A4, could be executing simultaneously. If a thread is executing in queue A1, no thread can access A2 until the thread in A1 exits; similarly for queues A3 and A4. This synchronization level should be selected when configuring STREAMS components that do not share state data outside of the component instance. Components that meet this criteria can, and should, also run with SQLVL_QUEUE.

Notice that both SQLVL_QUEUE and SQLVL_QUEUEPAIR permit separate simultaneous threads of execution in different instances of the same component, even different instances in the same stream. The two levels differ in that SQLVL_QUEUE permits a distinct active thread in each of the read-side and write-side queues of the component (both queue A1 and A2 at the same time), whereas SQLVL_QUEUEPAIR permits an active thread in either the read-side or write-side queue, but not both at the same time (queue A1 or A2, but not both at the same time).

This level most closely approximates single-thread STREAMS execution. Only a single active thread is permitted in any queue of any instance of the component. In Figure 13, if Component A has SQLVL_MODULE synchronization, only one thread can execute in any queue--A1, A2, A3, or A4--at any one time.

Designate SQLVL_MODULE synchronization if you are porting STREAMS components of the following types:

• STREAMS components having instances that share significant amounts of volatile data between themselves. When you configure components with SQLVL_MODULE, a bottleneck can occur when traffic flow is heavy across multiple instances (because only one queue is executing at a time). However, there are many cases of components handling very low traffic; the effort to rewrite those components to avoid data sharing may not be worthwhile.

• STREAMS components whose code architecture is unknown. You can safely configure the component with SQLVL_MODULE synchronization and modify it later with a finer level of synchronization, if you decide that the effort to understand and rewrite the code is worthwhile. Using SQLVL_MODULE may result in the same kind of traffic bottleneck as above, a trade-off faced in lieu of rewriting the code.

This is the most restrictive (and most rarely used) level of synchronization. It is used by a family of cooperating components to ensure that only a single queue in any instance of any component in the family is being executed at any one time. In Figure 13, if A and B are members of the same family of cooperating components and both are installed with SQLVL_GLOBAL synchronization, there may only be one thread executing in queues A1, A2, A3, A4, B1, B2, B3, or B4 at any one time.

A STREAMS component that shares data only during an open or close procedure would normally require the synchronization level SQLVL_MODULE. However, WindNet STREAMS allows you to take advantage of the greater flexibility in SQLVL_QUEUE or SQLVL_QUEUEPAIR when installing into WindNet STREAMS components that execute open and close procedures.

In WindNet STREAMS, regardless of the assigned synchronization level, all open and close procedures execute in a temporary level of synchronization equivalent to SQLVL_MODULE. During an open or a close, this special synchronization status supersedes settings of SQLVL_QUEUE or SQLVL_QUEUEPAIR. Synchronization of open and close procedures by WindNet STREAMS permits exclusive access to global data structures shared by component instances without forcing normal put processing to occur at the SQLVL_MODULE level.

For example, when opened, you may want a STREAMS component to link its instance data into a global list of component instances. In an open routine, the instance is guaranteed that it is the only active one of this component; therefore it's data can safely be linked to a global list without calling any special locking routines.

If a STREAMS component must invoke strmSleep( ) during an open or a close, WindNet STREAMS permits other instances of this component to execute at the configured synchronization level. When the STREAMS component wakes up, WindNet STREAMS ensures that the special synchronization of an open or close procedure is in effect before control is returned to those routines. The strmSleep( ) and strmWakeup( ) routines emulate UNIX sleep and wakeup functions. For more information, see the related manual entries.

For more information about the synchronization of instances of different STREAMS components, see §4.1.3.1 Writer Buddies.

Synchronization for put and service procedures is not the same as that provided for open and close procedures. When writing shared data using put and service procedures, synchronization must be coordinated between STREAMS instances. It is better to avoid sharing data across component instances to avoid incurring synchronization overhead. In cases when synchronization is necessary, WindNet STREAMS provides two ways to proceed.

The first method uses strmSyncWriteAccess( ) to control instances through the execution of the entire put or service procedure. Instances in single or multiple components that can be synchronized using this routine must be writer buddies (see §4.1.3.1 Writer Buddies).

The second method uses VxWorks semaphores to synchronize instances only at necessary sections of code in the execution of the put or services procedures, thus providing a finer resolution to synchronization.

A writer buddy is an instance of a WindNet STREAMS component that can gain exclusive access to a resource in a multi-threaded environment like WindNet STREAMS because of its association with other instances. Instances of a single component are designated writer buddies by default upon installation in WindNet STREAMS. Instances of different components can also be made writer buddies manually during installation by calling strmDriverAdd( ) or strmModuleAdd( ), see §4.4 Configuration of STREAMS Drivers and Modules in WindNet STREAMS.

It is this association of instances as writer buddies that ensures the execution of an open or close by only one member instance at a time; see §4.1.2 Synchronization During Open and Close Procedures).

Writer buddies can exist across multiple STREAMS components. Referring to Figure 13, Components A and B can be associated as writer buddies. This association implies that when an instance of Component A is opened or closed, all other instances of A and B will be locked out while the new instance of A executes its open routine. So, the special synchronization of open and close procedures presented in §4.1.2 Synchronization During Open and Close Procedures applies to both components. The cross-component association of writer-buddy instances of Components A and B can be established when the components are installed in WindNet STREAMS; see §4.4 Configuration of STREAMS Drivers and Modules in WindNet STREAMS.

Figure 14.   Writer Buddies

Component instances that are members of a writer buddy set can also be synchronized to have exclusive access to shared resources during put and service procedures by invoking the routine strmSyncWriteAccess( ); see §4.1.3.2 Updating Shared Data with strmSyncWriteAccess( ).

The strmSyncWriteAccess( ) routine coordinates multiple instances of a single component, as well as those in different components, so they can access shared resources one at a time. It is important that all instances synchronized by strmSyncWriteAccess( )already be designated writer buddies before calls are made to the routine.

The routine strmSyncWriteAccess( ) allows a single instance of a writer buddy set to gain exclusive access to a shared data structure from a put or service procedure. This mechanism works asynchronously, that is, strmSyncWriteAccess( ) does not necessarily execute upon its invocation. The caller of strmSyncWriteAccess( ) must provide:

1. A pointer to the routine responsible for updating the data structure.

1. A pointer to the STREAMS queue that needs to be acquired exclusively.

1. A pointer to a STREAMS message to be given to the routine passed to strmSyncWriteAccess( ).

The following code fragment illustrates the use of strmSyncWriteAccess( ):

int xxput(pThisQueue, pThisMsg)
{
    /* preliminary code not needing exclusive access to data structure.*/
    ...
    strmSyncWriteAccess (pQueue, pMsg, functionUpdate);
    /*
     * Either no code after call to strmSyncWriteAccess(), 
     * or code not depending on call to functionUpdate()
     */
    ...
}

void functionUpdate(pQueue, pMsg)
{
    /*
     * Code from xxput() requiring exclusive write access goes
     * here.
     */
    ...
}

This mechanism allows readers of shared data to access the data without special synchronization. The overhead of locking out all readers when the shared data structure needs to be changed is handled by strmSyncWriteAccess( ). This routine operates efficiently when a small number of instances of a STREAMS component are executing. The overhead of locking out instances increases in proportion to the number of active writer buddies. In a scenario having a large number of active instances, a more refined locking mechanism is available to lower overhead; refer to §4.1.3.3 Updating Shared Data Using Native VxWorks Synchronization.

WindNet STREAMS makes it possible to use VxWorks semaphores in STREAMS put and service procedures, with the following restrictions:

• Semaphores can be used only in STREAMS components that have been installed with a synchronization level of SQLVL_QUEUE or SQLVL_QUEUEPAIR.

• Semaphores can be held only in a single put or service procedure. It is important that semaphores not be held across putnext( ) calls; deadlocks can result.

• Semaphores used by STREAMS components in WindNet STREAMS must be created with the option SEM_DELETE_SAFE.

For a general discussion of VxWorks semaphores and how to use them, see the VxWorks Programmer's Guide: Basic OS.

STREAMS components executed in the WindNet STREAMS environment must adhere to the following rules in order to allow for synchronization:

1. The STREAMS routine putnext( ) and put procedures in STREAMS components can be called with any queue argument (called q). If the queue passed as the first argument to putnext( ) is the current queue (as would be normal for putnext( )), WindNet STREAMS synchronization prevents q and q->q_next from being changed by another execution thread. If, however, the queue argument is some other queue, it is the component's responsibility to assure the validity of the queue passed to putnext( ) and put( ), as well as the corresponding q->q_next field for putnext( ), that is, to validate that the pointer q and pointer q->q_next do not reference a queue already closed.

For example, a section of code outside the STREAMS framework wishes to call a put procedure to pass a STREAMS message into a component in the middle of a stream. This is a perfectly valid operation, and WindNet STREAMS synchronization assures that it will happen properly. However, WindNet STREAMS cannot detect whether the target stream is in the middle of a close or whether the target queue argument is stale. It is the responsibility of the component's close routine to communicate with the non-STREAMS entity and to receive a positive acknowledgment back from the entity before proceeding with the close. Only through this process can the passed queue argument be guaranteed to be valid.

1. STREAMS utilities that manipulate the message queue (putq( ), getq( ), insq( ), etc.) are valid only when the queue argument refers to the current queue. Thus, it is only safe to call message queue routines from within an open, close, put, or service procedure, and only if the q argument is within the same synchronization scope as the q argument to the procedure itself¹ . In particular, a component cannot invoke the callback routines bufcall( ) and esballoc( ) from a timer, since these routines have no queue context. This restriction can be overcome by passing a queue as part of the callback routine's argument. The callback routine can then call a put procedure on this queue argument, passing a private message which the put procedure can process.

1. Device drivers may safely call putq( )with the queue argument being the read-side queue of the bottom component on a stream (a STREAMS device). Calling putq( ) from a driver is a correct method for injecting a message into a stream. This can be done safely from interrupt context: WindNet STREAMS automatically detects this use of putq( ); it returns immediately after deferring placement of the message to the queue to a background thread. This thread then invokes the STREAMS device read-side service routine to process the message.

1. In the WindNet STREAMS implementation of freezestr( ), unfreezestr( ), qprocson( ), and qprocsoff( ), these routines are stubs which simply return. These routines are required by multiprocessor versions of SVR4 to assure single-thread execution for certain critical STREAMS activities, such as open and close processing. The synchronized access which these routines seek to provide is automatically assured by the various utilities in WindNet STREAMS, provided the rules listed in this section are followed.

1. Direct manipulation of the q_flag field of the queue structure is not permitted. Although the UNIX System V Release 4 Programmer's Guide: STREAMS prohibits direct manipulation of the q_flag field, some components violate this rule and are still able to run in standard SVR4 environments. Such manipulation does not work on WindNet STREAMS and may even cause system crashes.

Occasionally, STREAMS communication paths are unorthodox and unsupported by most STREAMS environments. You can develop workarounds by altering the queue structure field q_next to achieve any of the sample configurations in Figure 15. In a single-threaded environment, manipulating q_next is permitted; however, in a multi-threaded environment, like WindNet STREAMS, changing q_next is prohibited. WindNet STREAMS, however, enables you to build unorthodox communication paths by providing routines that synchronize access to flow control pointers.

Figure 15.   Examples: Non-Standard q_next Linkages

For example, to build a STREAMS pipe configuration, as shown in Figure 15, the q_next write pointer of each stream must connect to the read queue of the other stream. This process of altering the q_next pointer is called welding. WindNet STREAMS provides the strmWeld( ) routine for setting the q_next pointer safely. The routine strmUnWeld( ) returns q_next to NULL.

The strmWeld( ) routine executes asynchronously to the caller; therefore it takes an optional callback routine as an argument and two optional arguments to be passed to the callback routine. In addition to these parameters, strmWeld( ) can be passed two destination queue pointers set to the queue pointers specified in the source queue parameters.

To continue building the streams pipe, use strmWeld( ) as follows:

strmWeld (pWrQA,pRdQB,pWrQB,pRdQA,NULL,NULL,NULL);

Such a call results in the pWrQA field being set to pRdQB. Similarly, the pWrQB field is set to pRdQA. Use strmUnWeld( ) to undo the link between Components A and B by setting the q_next fields pWrQA and pWrQB to NULL, as follows:

strmUnWeld (pWrQA,pWrQB, NULL, NULL, NULL);

In both preceding examples, the callback routine and its two optional arguments are set to NULL. Use the callback routine if the STREAMS component requires notification that the operation has completed.