The detour began last April as I was writing my PID control algorithms. The algorithms themselves are not difficult but they do require parameterization. I knew that didn't want to tweak the values, load up a new version, try it, and iterate. I instead wanted real-time control of the parameters. I also wanted a log of all commands (stop and balance, move to the left, etc.), sensor data (accelerometers, gyroscopes and rotary encoders) and motor direction (PWM pulse widths) in one central place. Having this data would not only make the internal workings more transparent, but it would allow post-run analysis and support comparison of different algorithms and/or parameterization. Since the sensor are sampled 50 to 100 times a second, I needed an approach that could handle these speeds over many minutes. Ideally it should handle this sampling rate for extended periods of time.
My ideal vision for this logging and control system is a high speed pub/sub system supporting many individual tracks. I see it as a 2d graph where the Y axis represents each pub/sub channel and across the X axis is time marching on. Individual components of Godfrey can read and write events in any track of interest. In the image above, I show a command track that is written by some higher level function, the PID algorithm takes the command plus the data from the gyroscopes and accelerometers and it outputs the speed and direction of the left and right motors.
Having a pub/sub at the core of Godfrey has other impacts in the design. I have spent the last many months fleshing out the details to determine what kinds of hardware to use, how to develop the distributed timeline system, what languages to write my major components in, and what planner technology to use. This has lead to examination of many different hardware and software systems including: Arduinos, PandaBoards,Raspberry Pis, BeagleBone Blacks, C/C++,
Erlang, Clojure, Go, Python, Ruby, UIMA, Kafka, Redis, ROS, ChatScript, HAP and POSH.
Future posts will have to discuss the details of the full design. However, some of my conclusions are:
Arduinos are amazing devices. I love their low cost, all those lovely digital and analog I/O pins and the great built-in library support for I/O devices. But in my opinion their networking is quite not as pleasant as the rest of the system. Any low cost networking options would require me to roll-my-own hardware and software stack or I can use a Ethernet/WiFi shield with its so so network library. So instead of using Arduinos I have decided to use BeagleBone Blacks. They have similar I/O capability as Arduinos, but have a much more powerful processor and support a full Linux stack with its networking. They are also modestly priced. It is truly a joy to be able to write normal Unix programs and have it control low level I/O. It is also very cool that the same platform has the power to run complex speech analysis, image processing, and AI software on the same platform.
The use of BeagleBone Blacks (BBB) open up the languages I can program in. I know that there will be some C/C++ in Godfrey to link to libraries I will be using like PocketSpinx, Flite, and OpenCV. However, for higher level reasoning tasks and the robot planner I plan to use Go. I benched its speed against Clojure (another language I love) and Go is substantially faster on the BBB. Pragmatically, I know that I should have chosen Python because it is already supported by BBB and is commonly used by the robotics community. But for some reason I have never enjoyed programming in Python. In the end, Godfrey is my design and I just don't want to face the prospect of writing a lot of Python.
My experience using Redis, DynamoDB and SQS has convinced me of the power of simple and lightweight network APIs against simple data services. Store the data blocks in some common data exchange format such as JSON and you have a very power interconnect strategy. This strategy allows different components, running on different servers, written in different language, built in totally different epochs of time, by teams of different programming ability to all peacefully work together. As much as I wanted to like and use ROS, I find its programmatic API rigid, complex, heavy. It is in stark contrast to these competing network API services. This complexity and heaviness means that ROS only supports a limited number of languages. I have decided that I want to use a Redis-style network service supporting a pub/sub data model that blends the best of Redis and Kafka's.
I had previously believed I would construct a HAP-style reactive planner to control Godfrey. However, in the last months I have discovered POSH. I have been amazed at the simplicity of its overall design while supporting the rich features commonly found in other reactive planners. POSH also is very well suited to systems that connect complex self-contained competencies such as the balancing system, speech recognition, etc. with the POSH planning system. POSH is also compatible with my desire to use a distributed pub/sub system. It will take a lot longer post to explain what I see in POSH and how I plan to use it.
I have been doing many experiments to test out these ideas and how well they work together. To speed up end-to-end development I have created a experimental testbed on top of an iRobot Create. It uses the Create for movement, collision detection, and power. Added to it are some cheap switching power supplies to convert the unregulated 18 volts to 5V, a BeagleBone Black, serial port level converters, and a Wi-Fi Ethernet Hub. All the hardware is up and running under software control over my home WiFi network. I am now working on the software architecture discussed in this post. Eventually I will adding another BBB, a USB camera, mic, and speaker to support speech and image processing.
I will be outlining my progress in future posts.
