2011-10-13

Ach Track

One thread for sockets and XML-RPC
- Multiplexes file descriptors with poll/select
- Puts messages into queue
One thread to process messages
- ros::spin()
- processes the messages queued up by the socket thread
Other threads
- Can register specific callbacks with specific threads
- Can start up other threads as desired

Inital Code
To minimize latency, it is desirable to minimize context switching.
- Thus, the thread that pulls the message out of the channel should be the thread to handle (invoke the callback on) that message.
- Callback is stored in the per-topic subscription object
- ACHROS could pull that callback out of the subscription object and
  - invoke it directly.
The File Descriptor / Condition Variable Disconnect
- ROS topics accessed via file descriptors which are waited on in a
  - poll/select
- Access to ach channel is mediated by a mutex/condition variable
  - which subscribers will wait on
- POSIX offers no way for one thread to wait on multiple condition
  - variables
- Two options to access Ach channel
  - (1) Poll the channel
  - (2) Wait (on condition variable) for the channel
  - Both used by GT-Humanoids, waiting when process has only couple
    - channels, polling when there are many channels.
SVN
- Electric already branched out
- Work off of trunk
- Even/Odd Stable/Unstable version numbering
ROS REP 106 describes a polling API for ROS subscriptions. This
- might be a good fit for Ach's model which supports both waiting and polling for messages.

Project: multi-robot mapping, teleop with video and laser data to
- operator
Problems when out of WIFI range, messages queue up then burst when
- back in WIFI range, even with UDP.
Bufferbloat: the WIFI layer detects packet loss (no ACK of WIFI
- packet) and retransmits. This causes large and unacceptable latency.
Used Linksys Router and 1w (WIFI max power) Ubiquiti router.
Sending large UDP messages works fine when extra network capacity

Lincoln developed network monitor to track health of connections, ie
- ping times
- Communicates data as ROS messages
- nodes can then dynamically adjust ie jpeg compression based on
  - available bandwidth
Possible to release the Network Manager
- 45-day review process at LL
Want some way to request a certain amount of network bandwidth
- LL usings trickle/IPtables to enforce contracts
Perhaps ROS needs a "Network Master" to manage QoS
Question: what are the IP and Kernel layer QoS options?

Some Smart compression for point clouds
- Uses deltas, so good for static scenes but bad for mobile robots
Naive GZIP stream compression: not good enough
WG sends point clouds over WIFI
- low frequency
- takes over network

Joe Romano, PhD at UPenn, interest in haptics and feedback control
Brian Gerkey, Director of Open Source at Willow Garage, did lots of early ROS development
Azamat Shakhimardanov, PhD at Bonn, interest in control and middleware
Mike Stilman, Professor at Georgia Tech, interest in planning and control
Neil Dantam, PhD at Georgia Tech, interest in formal methods for control
Patrick Mihelich

Different applications probably require different transport semantics and implementation tradeoffs.
Appropriate to support multiple transports, and ROS provides some infrastructure to do this.
Implement multiple transports. Community can try and decide which are appropriate.

TCP: stream backs up when many packets dropped
UDP: multi-packet messages (ie big images) get lost when any packet in the message is dropped
If initial connect to publisher fails (due to dropped packets), subscriber assumes publisher is dead.

Real-Time control needs minimal latency
Vision needs minimal copies and low latency
Vision probably will still requires each process to copy the message to its own address space.

Supports single-slot buffer or queued messages.
Multiple subscribers will each "remove" a message from the queue, preventing others from seeing it.
Handles serialization also via XDR.
Does networking, reliable/unreliable message transport.

Unique semantics for message-passing.
Always access newest message. Older messages on best-effort basis.
Distribution to n subscribes require n+1 memcpy's. (Point-to-point sockets generally take 2*n memcpy's due to in-kernel buffer)

Investigate wireless transport needs.
Integrate Ach as a roscpp transport. Maybe stretch internal API. Try not to change external API.

Everybody: Consider transport-over-WIFI needs and investigate options.
Brian: Poll ROS users on their needs for wireless transports.
Neil: Get Ach building as a ROS package. Check roscpp code to for how to integrate Ach.
Joe: Test a ros-integrated Ach.
Azamat: Review API differences between ZeroMQ, RCSLib, and Ach