Channel User Manual
Yigong Liu (2/25/2006)
1. Design and Architecture
1.1.
Application
domains characteristics
1.2. Design
aspects of message passing and event dispatching applications
1.3. Solution
2. Sample applications and generic development procedures
2.1. Simple send/recv applications
2.2. Generic
procedures
3. Frameworks classes
3.1. Channel
centered framework classes
3.2. Connector
centered framework classes
4. Channel specialization
4.1. Use
application specific IdType
4.2. Use
application specific Routing Algorithms
5. Connector specialization
5.1. Transport
5.2. Filters,
Translators
6. Message/event handling
6.1. Message
creation
6.2. Message
destroy
6.3. Message
forwarding and distribution
7. Install and build
1. Design and
Architecture
1.1. Application
domains characteristics
Therer is a common theme in today's software
applications: interactions - they are interacting with human, with the
external environment and with each other. Interacting software
applications are mostly asynchronous (possibly distributed), driven by
events or messages. Most of them follow one of the following
models.
Large asynchronous systems involving
multiple resources (processes, computers...) mostly employ distributed
message passing design. Various application domains exhibit different
requirements and tradeoff for performance, space, and maintainance:
- embedded systems prefer designs with minimal space and
computation overhead because of limited hardware resources and
performance requirement; normally integer (or sub-integer, such as
bit-fields) are used as routing id and simple POD struct are used for
passing messages.
- desktop and e-commerce applications prefer designs with more
expressive/meaningful routing data structures and algorithms to deal
with more complicated human world.
event dispatching systems are applied
in single threaded systems for a flexible system structure to handle
asynchronous external events. The examples are :
- libraries for handling network/IO events (eventlib, libaysnc)
- GUI libraries and frameworks (java GUI event dispatching, MFC
message_map structures).
1.2. Design aspects of
message passing and event dispatching applications
Although message passing applications and event dispatching
applications appear quite different, internally they share a common
core. The design of most message passing and event dispatching systems
can be separated into the following "orthorgonal" aspects:
- Statically configurable aspects:
- approriate message/event Id types
- routing algorithms
- synchronization strategies for routing data structures (for
multithreaded or single-threaded applications)
- Dynamically configurable aspects:
- namespace management, scope control (Id visibilities)
- remote connections
- channel messages/Ids translators and filters
- message memory management, marshaling and demarshaling
An ideal solution should allow flexible mixing and matching various
implementations of the above aspects to create a best-fit message
passing or event dispatching facility customized to the specific
application characteristics.
1.3. Solution
Channel is a C++ template framework to facilitate users
to create the best-fit publish-subscribe message passing or event
dispatching architecture:
template
<
class Id_Type,
class Id_Trait = IdTrait<Id_Type>,
class SynchPolicy = ACE_MT_SYNCH,
class AllocPolicy,
class Router = MapRouter<Id_Type, Id_Trait, SynchPolicy, AllocPolicy>
>
class Channel
By
specializing the above template with appropriate message id types and
routing algorithms, Channel can
support table/linear message routing, hierarchical
routing and (in future) associative routing (Linda style) at the demand
of user, making the message passing
and event dispatching as straightforward to use as STL data structures.
Channel is built on top of ACE (Adaptive Communication Environment) to
achieve platform independence.
2. Sample channel based
send/recv applications
2.1.
Simple send/recv applications
- simple send/recv using hierarchical routing based on trie (prefix
tree)
- simple send/recv using linear routing based on table/map
2.2. Generic procedures
The following are the generic procedure to
develop channel based applications:
<1>
Partition the application into threads and/or callbacks. Clarify the
messaging/event interface among threads and/or callbacks: what
messages/events will be passed.
<2> Specialize channel template for
the application. Choose appropriate Id_Type, Id_Trait and Routing
algorithms
<3> Define the message/event ids
and data structures
<4> Application logic flow:
<A>
create an instance of specialized channel
<B> connect channel to remote
channels for distributed applications, using proper transport
mechanisms such as tcp socket, unix domain socket, or others.
<C> create channel publishers and
subscribers
- create instance of Port for asynchronous
message passing, subscribe to interested message/event ids, publish
message ids to appropriate scopes
- create instance of Callback for
synchronous callbacks and subscribe
to interested message/event ids
- create instance of Source as publishers
and publish message/event ids to appropriate scopes
<D> start application logic/threads to send/receive/process
messages/events
3. Frameworks classes
There are 2 co-operating sub-frameworks inside. The first is centered
around class Channel for setting up publish-subscribe namespace and
message/event routing. The 2nd is centered around class Connector for
remote channel connection, transport, and message
marshaling/demarshaling. The joint among these 2 sub-framework is class
Interface which is a publisher/subscriber in Channel sub-framework and
connection endpoint in Connector framework.
3.1. Channel centered
framework classes
Channel centered sub-framework involves the following major classes:
- Channel: a simple wrapper
of all channel related types
- IdType: primitive types,
POD struct and classes can be used as message/event id - the "key" for
publish, subscribe, and routing. Id type classes for table/map based
routing should define the following operations:
- bool operator< (const IdType &id) const
- bool operator== (const IdType &id) const
- bool operator!= (const IdType &id) const
- IdTrait: defines the
attributes and operations necessary for associated routing algorithms:
- static bool match(const IdType &id, const IdType &id2)
const
- Predefined channel system internal message/event ids for
channel connection, disconnection, publication, subscriptions, etc.
- Router: implementation of
table based routing and hierarchical routing
Use STL map as the data structure to
implement a table (key matching) based (linear) routing algorithm.
Use a tree data structure similar to
"trie" to implement hierarchical (partial pathname / prefix matching)
routing algorithm
- Source: the publisher,
where messages are published and sent out. The parent class of all
publisher classes such as Port and Interface.
- Destination: the
subscriber, where messages are subscribed and consumed. The parent
class of all subscribers such as Callback and Port:
- Callback:
parent class of synchronous callback
- Port:
play the role of both source and destination for
asynchronous message passing. Peer threads can use it to send and
receive messages. It contains an internal message queue for buffering
- Msg: a generic message
container/holder used in Channel for message passing and/or event
dispatching. refer to section 6 for more details.
Channel centered framework classes are mostly platform
independent. The platform dependent parts are thread synchronization
primitives and memory alloc/free; which has been (or will be) wrapped
in 2 template policy classes: SynchPolicy and AllocPolicy. Right now
ACE's SynchTrait is used for SynchPolicy. While instantiating
Channel template for applications, user can specify ACE_MT_SYNCH or
ACE_NULL_SYNCH for multithreaded or single-threaded applications.
3.2. Connector centered
framework classes
Connector centered sub-framework includes the following major classes:
- Connector:
the main class to integrate all
channel
connection classes, maintains active connections to remote channels.
Internal threads are spawned to read from and write to network
connections.
- LocalConnector:
a special version of Connector for connecting
channels inside the same process, very light weight, just pointer
passing, no internal thread or queuing.
- Interface:
the joint between Channel
framework and
Connector framework. It is a "proxy" for remote channel, playing the
role of both publisher and subscriber in the local channel.
- Binder:
contains filter and translator,
control
the namespace merge happening at a interface "binding" to
specific remote channel.
- Filter: control what messages/events can flow thru an interface
in both directions.
- Translator: control where in the local channel namspace, the
namespace from remote channel will be relocated.
Application code can create binders
with customized filters and
translators to implement application specific policy and register it
with connector for an external address.
- Transport: provide
Connector with transport mechanism.
- Marshaler:
for distributed channel application with complicated message data
structures or involving heterogeneous machine architectures, a
marshaler class should be defined for application messages by
inheriting from class Marshaler, whose marshal() and demarshal()
methods should be implemented using a common network data format (such
as CDR). The marshaler classes should be registered with Connector
using Id as index.
4. Channel specialization
4.1. Use application
specific IdType class
To make sure application specific Id type
working with Channel, class Id_Type and class Id_Trait should define:
- the operators and functions required for routing data structure
and computation
- 8 system internal messages/event ids
For normal systems based on table/linear routing,
Channel specialization is as simple as Channel<Id_Type>.
4.2. Use application
specific Routing Algorithm
To use application specific routing algorithms with Channel framework,
the application router class should provide the following methods:
Status publish_msg(IdType t, PubSub_Scope s, Source
*src)
Status unpublish_msg(IdType t, PubSub_Scope s,
Source *src)
Status subscribe_msg(IdType t, PubSub_Scope c,
Destination *dest)
Status unsubscribe_msg(IdType t, Destination *dest)
Status route_msg(Msg *msg, Member_Type src_type,
PubSub_Scope pub_scope, timeout=0).
For the performance of message/event routing inside Channel, Source and
Destination will maintain its local namespace and perform id
validation locally. Possibly application specific router will implement
a very different namespace which may require customized version of
Source and Destinations. A namspace type enum and template partial
specialization is used to make sure correct versions of Router, Source
and Destination are used together.
5. Connector
specialization
5.1. Transport
5.2. Filters,
Translators
6. Message/event handling
6.1. Message creation
Many message passing systems require that messages are
dynamically
allocated and message class/structures must inherit a base class with a
virtual destructor so that all messages can be cleaned up properly.
In Channel, messages creation are flexible:
- different data type:
- a class/POD struct
- an array
- different memory type
- dynamically allocated
- a static entity (global or on stack)
Messages in Channel don't need to inherit from a common base class.
Internally application messages will be contained inside the generic
message holder: class Msg which will avoid message copying and destroy
message correctly in the end.
6.2. Message destruction
Message destruction in channel applications are involving:
- because message creation variety (diff data type, memory type, no
common base class)
- may involves different runtime heaps:
since channels publishers &
subscribers may be threads or callbacks in different DLLs, messages may
be created in one DLL and freed in another
Solution: use MsgFreeCallback:
- to correctly free messages memory to the correct heaps, pass a
MsgFreeCallback from source DLL to destination DLL and free message
using this callback
- for mostly used class/POD struct messages, use template to get
the message type info and auto-generate correct message free callback
- if message data is an array, need to pass in message size
explicitly, and use DefaultMsgArrayFreeCallback (which call delete[])
- use no-op free callback, if message data are static or on stack,
or if sender want to keep & reuse the messages and prevent
receivers from deleting it.
6.3. Message forwarding
and distribution
Since channel is pub/sub channel and there are possible multi
subscribers to the same message type in different threads, we need a
way to
avoid copy/duplicate messages to send to different subscribers.
Solution: using reference
counting for all subscribers to share the
same copy of message
subscribers must obey the following convention for sharing to work:
- get the payload data by message->data()
- cast it to proper type before use
- delete message, DO NOT delete payload data directly, payload will
be properly freed after refrence-count is 0.
7. Install and build
7.1. setup
1.1 install the latest version of ACE (Adpative
Communication Environment)
The Channel lib is built on top of
ACE for platform abstractions
1.2 download the Channel lib (.tar.gz file) and unpack it
under $CHANNEL_ROOT
1.3 set up env
add the following to
.bashrc:
#for
Channel lib
export CHANNEL_ROOT=
export CHAN_ROOT=$CHANNEL_ROOT/channel
export
LD_LIBRARY_PATH=.:$CHAN_ROOT/lib:$LD_LIBRARY_PATH
export PATH=$CHAN_ROOT/bin:$PATH
#for CVM
export CVM_ROOT=$CHANNEL_ROOT/cvm
export
LD_LIBRARY_PATH=.:$CVM_ROOT/lib:$LD_LIBRARY_PATH
export PATH=$CVM_ROOT/bin:$PATH
7.2. build:
cd $CHANNEL_ROOT
make
7.3. run samples
- in console 1:
cd $CHAN_ROOT/examples/hier_text_recv
hrecv 6666
- in console 2:
cd $CHAN_ROOT/examples/hier_text_recv
hsend 6666