The Performance of Seperating Control and Video Display Processes in UAV::Pilot

I’ve been running some crude benchmarks of the UAV::Pilot video timing. As I went over in my last post, I’m planning on having the video be read from the network in one process, and have it piped out to another process for decoding and display.

I added logging statements that show the exact time (using Time::HiRes::gettimeofday()) that a video packet comes in, and then another log for when we display it on the SDL window.

The first benchmark used the existing uav_video_display that’s in the UAV::Pilot distribution, reading from the file ardrone_video_stream_dump.bin. This file is in the UAV::Pilot::ARDrone distribution and is a direct dump of the network stream from an AR.Drone’s h.264 video port. It’s primarily used to run some of the video parsing tests in that distro.

I found that on my laptop, there was a delay of 12.982ms between getting the video frame and actually displaying it. At 60fps, there is a delay of 16.667ms between each frame, so this seems quite acceptable. The AR.Drone only goes up to 30fps, anyway, but it’s nice to know we have some leeway for future UAVs.

I then implemented a new script in the UAV::Pilot::ARDrone distro that read the same video frames from STDIN. I had planned on doing this with the same file noted above, like this:

cat ardrone_video_stream_dump.bin | ardrone_display_video.pl

But this ended up displaying only the last frame of video.

My theory on why this happens is that we use AnyEvent for everything, including reading IO and telling SDL when to display a new window. Using cat like that, there’s always more data for the AnyEvent->io watcher to grab, so SDL never gets a chance until the pipe is out of data. At that point, it still has the last frame in memory, so that’s what it displays.

I tried playing around with dd instead of cat, but got the same results.

So I broke down and connected to the actual AR.Drone with nc:

nc 192.168.1.1 5555 | ardrone_display_video.pl

Which did the trick. This does mean that the results are not directly comparable to each other. We can still run the numbers and make sure the delay remains insignificant, though.

And indeed it did. It averaged out to 13.025ms. That alleviates my concern that using a pipe would introduce a noticeable delay and things can go right ahead with this approach.

Announcing: The Great UAV::Pilot Split

The main UAV::Pilot distro was getting too big. The WumpusRover server had already been spun off into its own distribution, and now the same is happening for many other portions of the system.

Reasons:

  • UAV::Pilot hasn’t been passing CPAN smoke testing because of the ffmpeg dependency. Splitting that off means the main distro can still be tested.
  • There’s no reason to get the WumpusRover stuff if you just want to play with the AR.Drone
  • General aim towards making UAV::Pilot into the DBI of Drones

UAV::Pilot itself will contain all the basic roles used to build UAV clients and servers, as well as what video handling we can do in pure Perl.

UAV::Pilot::Video::Ffmpeg will have the non-pure Perl video decoding.

UAV::Pilot::SDL will take the current SDL handling, such as for joysticks and video display.

UAV::Pilot::ARDrone and UAV::Pilot::WumpusRover will take the client end of things for thier respective drones.

I expect the first release to have interdependency issues. I’ve tested things out on a clean install, but I still might have missed something. I’m going to be watching the CPAN smoke test reports closely and filling in fixes as they come.

UAVs and FOSS — AR.Drone

This is the first in a series of articles about the state of FOSS software in terms of UAVs. We start with the place where I have the most experience, the Parrot AR.Drone.

On the client side, Parrot has released a complete SDK for controlling the AR.Drone. It includes support for the control protocol, the navigation protocol, and the video streams (which is handled by ffmpeg).

However, many people choose to make a clean implementation based on the SDK docs. This includes my own UAV::Pilot, and also NodeCopter, an implementation for Node.js. Both of these are under a BSD-style license. The parts of UAV::Pilot that interface to ffmpeg are LGPL.

Control is done over WiFi. It will create its own ad-hoc network by default. With some trickery, you can connect it to an open access point, though it doesn’t seem to support WPA authentication.

The on-board electronics for the AR.Drone has a Linux system with Busybox, which you can telnet into. This seems to be mostly useful for debugging the commands sent over the network, or configuring it to connect to an open access point. Hacking it to control external devices via USB seems like it should be possible, though I haven’t been able to find anybody who actually accomplished this. Just a lot of people talking about it. C’est la vie.

Version 2.0 of the AR.Drone includes front and bottom cameras, which can stream 720p video. They’re on par with cheap webcam quality.

As I’ve complained about before the AR.Drone’s SDK doc is rather poorly documented. The control protocol is mostly fine, but there are a lot of missing details about the nav data and video protocols. The information you need is spread out on the dev fourms, and it just shouldn’t be that way. I’m also dealing with the multiconfig options, which requires getting the ACK bit off the nav data stream, which seems all backwards to me even if it was well-documented. The config should have been handled over a TCP stream, especially when multiconfig was implemented.

It is relatively cheap, though, at $300. A basic ArduPilot setup will be around $700 if you include the telemetry module and a cheap 6-channel radio. That setup wouldn’t even have a camera.

All things said, I think the AR.Drone is a good way to get started considering what’s out there right now. But it could definitely be improved.

UAV Basics — Types of UAVs

This will be the first of a series of articles about the basics of UAVs. Today we start with the different types out there.

The first is your standard airplane model. This one is the Rhino UAV, which is a project intended to help anti-poaching efforts:

Rhino UAV

The design is similar to all the model airplanes that have been out there for years. The only real difference is that the on-board electronics can be made sophisticated enough to fly to a destination without any human input.

The other kind of flying UAV is the helicopter type, though the traditional helicopter doesn’t seem too popular to build. The multipod design, especially quadcopters, seem more widespread.

CycloneCloseup1

There were a number of historical attempts to build a multipod, but the single-rotor helicopter became preferred in general aviation, despite the problem of the torque on the blades inducing it to spin. The reason seems to be that slight differences in motor speed, weight balance, and propeller shape tend to make multipods unstable. The pilot has to make constant corrections for this, and it becomes too much of a mental load. Meanwhile, the helicopter’s torque problem was solved with a simple rear vertical propeller, so everybody just did that.

Now we have cheap microcontrollers, gyros, and accelerometers for automatic stabilization. That takes the load off the pilot, making this a viable design. Quadcopters have particularly grown in popularity of late.

One variation I’m interested in is this 3-armed, 6-propeller design from 3D Robotics:

Tri Copter

The dual prop design lets it have high lifting capacity in a small package.

UAVs don’t just fly, though. They drive and swim, too. Google’s self-driving car is essentially a UAV. The ASV Roboat is an autonomous sailboat used for research into the endangered harbor porpoise in the Baltic Sea:

ASV Roboat

Most of what I’ll be working on is the flying variety, though.

UAV::Pilot v0.5 Released, Now Supports Real Time Video

UAV::Pilot, a Perl library for controlling the Parrot AR.Drone, has released version 0.5.

This was the big one. With the ffmpeg library installed, you can decode the h.264 video stream coming right off the drone, in real time. This is where UAV::Pilot starts to get really interesting, opening up applications like augmented reality games and object recognition.

Demo video:

I’m going to be taking a short break from working on UAV::Pilot for a while. After lazing about, I’ll start work on the next release, which will mostly be cleaning up the API and bugs.

Also, I’ll be giving a presentation on UAV::Pilot in September for the Madison Perlmongers Group, which I plan on recording.

UAV::Pilot v0.4 Released, Now Supports Video

UAV::Pilot, a Perl library for controlling the Parrot AR.Drone, has released version 0.4.

The big change for this update is streaming video to a file. Display in real-time is not yet
implemented–that’s coming in 0.5.

There are also some API changes in how event loops are initialized to make the naming more consistent. There will probably be another API change in the video handling in 0.5. Once I do a complete release, API changes will have a proper deprecation plan, but for right now I think it’s fine to change things willy-nilly.

UAV::Pilot v0.3 Released

Version 0.3 of UAV::Pilot has been released on CPAN. The major change in this version is an improved event loop based on AnyEvent.

AnyEvent is a very nice framework overall, but unfortunately doesn’t have quite the right syntax for UAV::Pilot’s purposes. Consider this plain-English description of a UAV flight:

Takeoff, wait 5 seconds, then pitch forward for 2 seconds, then pitch backwards
for 2 seconds, then land

In AnyEvent’s standard interface, things would work out something like this:

my $timer; $timer = AnyEvent->timer(
    after => 5,
    cb => sub {
        $uav->pitch( -1.0 );
        my $timer2; $timer2 = AnyEvent->timer(
            after => 2, 
            cb => sub {
                $uav->pitch( 1.0 );
                my $timer3; $timer3 = AnyEvent->timer(
                    after => 2,
                    cb => sub {
                        $uav->land;
                        $cv->send;
                    },
                );
                $timer2;
            },
        );
        $timer;
    },
);

$uav->takeoff;
$cv->recv;

All that indentation gets hairy. So I created an interface on top of AnyEvent, UAV::Pilot::EasyEvent, which is based on the Node.js event interface used by NodeCopter. Here’s how you’d write the same problem using EasyEvent:

my $cv = AnyEvent->condvar;
my $event = UAV::Pilot::EasyEvent->new({
    condvar => $cv,
});

$event
    ->add_timer(
        duration => 2000,
        duration_units => $event->UNITS_MILLISECOND,
        cb => sub { $uav->pitch( -1.0 ) },
    )->add_timer(
        duration => 2000,
        duration_units => $event->UNITS_MILLISECOND,
        cb => sub { $uav->pitch( 1.0 ) },
    )->add_timer(
        duration => 2000,
        duration_units => $event->UNITS_MILLISECOND,
        cb => sub { 
            $uav->land;
            $cv->send;
        },
    );

$event->activate_events;
$uav->takeoff;
$cv->recv;

Much better.

As an added bonus, having a good event interface means we can also support controlling UAVs through SDL joysticks. See bin/uav_joysticks in the distribution for details.

Announcing: UAV::Pilot v0.2

UAV::Pilot version 0.2 is now up on CPAN. The big change in this one is initilizing the navigation stream and parsing the data. I also whipped up this retro-videogame looking graphical output:

nav_output_sdl

The usability of the nav data is a bit limited at this point, because we need to grab the stream continously in a loop for display while also processing the flight commands. This is where an event system like AnyEvent will come in handy. You can try programming it around AnyEvent (or whatever event system you like) with what’s there right now.

The major work for the v0.3 release will be getting an event loop integrated into the library.

AR.Drone Nav Data

There are parts of the Parrot AR.Drone that are rather underdocumented, and one of those is getting the navigation data.  Once you activate the nav data, it doesn’t send the data directly to you. Instead, it sends it to the 224.1.1.1 muticast address.

Which is great and all. That means you can have one person controlling on their laptop and a bunch of other people watching the nav data on their own systems. But it’d be nice if their docs told you this.

There’s also a way to get it to unicast to your address, but they certainly don’t lay that out.

Anyway, the good news is, I can receive the AR.Drone’s nav data now, which is a big milestone for the 0.2 release.