paparazzi-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Paparazzi-devel] What's the state of ardrone2_vision?


From: Ori Pessach
Subject: Re: [Paparazzi-devel] What's the state of ardrone2_vision?
Date: Mon, 15 Dec 2014 08:20:40 -0700

I used the dsp binaries that came with ardrone2_vision, and simply replaced the rtp packetizer (rtpmp4vpay) with a filesink that had the location set to a file under /data/vision (which filled up the filesystem very quickly, of course.)

Basically, it was just 'dspmp4venc ! filesink location=/data/video/video.mp4v' or something like that, at the end of the chain. So the whole thing probably looked like this:

gst-launch v4l2src device=/dev/video1 ! videorate ! 'video/x-raw-yuv,framerate=10/1' ! videoscale ! 'video/x-raw-yuv, width=848, height=480 ! dspmp4venc ! filesink location=/data/video/video.mp4v

This gives you an MPEG-4 elementary stream, which you can then multiplex into an MP4 file using ffmpeg:

ffmpeg -i video.mp4v video.mp4

I would have preferred to simply have the pipeline produce an MP4 file, but the mp4mux element isn't included in the binaries provided with ardrone2_vision, so I'm working on recompiling the whole gstreamer package (and possibly updating it to something a little more recent than 0.10.0...) and so far it's been a pain. glib isn't exactly straightforward to cross-compile, for example. But it's getting there. 

My plan is basically:

1. Cross compile gstreamer and get all the parts I need to run on the drone (in-progress)
2. Create a simple paparazzi module to record video during flight and to control the recording from the flightplan
3. Figure out how to interface with the Parrot /bin/dsp/720p_h264venc_sn.dll64P and see whether I can adapt the dsp-gst gstreamer H.264 encoder element to use it.
4. Assuming that step 3 isn't a dead-end, modify the module to record H.264. 

Streaming video while recording shouldn't be difficult, but the poor results I've seen so far make me suspect that the version of gstreamer I'm working with might not be handling things very well when it comes to UDP streaming. I don't really know what the problem is, but I know that tossing RTP packets over a local network is usually more reliable than what I've seen gstreamer manage to do, so there's room for improvement there. 

I don't have a timeline for achieving any of this, so don't hold your breath... What's clear, though, is that since the Parrot firmware can deliver rock solid video performance at 720p and high framerates, the hardware is certainly capable of doing more than what we've seen so far with the vision framework. So there's that.

Ori

On Mon, Dec 15, 2014 at 1:15 AM, Christophe De Wagter <address@hidden> wrote:
This is great news!

I thinks a paparazzi module to detect USB filesystems and start recording should be easy. If you can tell us which image you loaded on the dsp and which gstreamer chain you used for recording.

-Christophe 

On Mon, Dec 15, 2014 at 2:59 AM, Ori Pessach <address@hidden> wrote:
Not if you want the result to be a normal MPEG-4 video bitstream, you can't... :)

I was able to encode a larger frame size, but I suspect that the codec itself (the dll64P file that runs on the DSP) has some limitations. Still, I was able to encode an 848x480 video at 10fps, and it seemed to work fine. It actually worked perfectly as long as I saved the video to the drone's flash filesystem (using a filesink.) Streaming it over UDP has a lot of issues, but since saving the video locally works perfectly I think that points to the UDP element in gstreamer as the problem. Either that, or the client side is broken, but I used both VLC and gstreamer as clients, and both behave in the same way, so I don't think this is a client issue. 

I'm still planning to get the H.264 encoder that's already on the drone to work with gstreamer, if that's possible. I think I understand enough of gst-dsp to at least give it a try. 

I also want to get the mp4mux working, so I can record MP4 files during flight, and a working RTSP server would be a nice thing to have, for my own twisted purposes. 

To make recording MP4 files more convenient than it currently is, it would be nice to have a module that can autodetect and mount a USB drive when one is inserted. It would also be necessary to have the recording module stop cleanly (so it can close the MP4 file and finish writing all the necessary boxes that make it playable.) The signal to do that would have to be tied to a flight event - probably something in the flight plan that happens after the drone lands. This is basically how Parrot handled recording - except that their firmware also looks for unclosed files and tidies them up, which I was actually really impressed to see. If any Parrot engineers are reading this - nice work! You saved a video from one of my more dramatic crashes, and saved me the work of trying to salvage the video myself.

Ori


On Sun, Dec 14, 2014 at 3:22 PM, Christophe De Wagter <address@hidden> wrote:

We were wondering why this dspmp4enc could not handle larger frames. We were thinking of a memory issue. Can't you send 4x 320 frames to represent 1 640 frame but allow smaller buffers?

On Dec 14, 2014 7:40 PM, "Ori Pessach" <address@hidden> wrote:
Thanks! This looks useful as a starting point for a more full featured implementation, but as it stands now it falls short of what I'm looking for. 

First, it only streams low resolution MPEG4 video over UDP, whereas I'm interested in storing 720p H.264 video on a USB stick while simultaneously streaming a lower quality stream to a laptop. Even the lower quality stream shouldn't be 320x240. That's kinda bad. 

There are many missing piece which I'm currently working on figuring out: the ISO-14496-12 multiplexer appears to be missing from the version of gstreamer that comes with ardrone2_vision, and there's no working H.264 encoder. The former is relatively easy to fix, while the latter seems to be a bit trickier. 

I'm basically trying to get to feature parity with the Parrot firmware, at least when it comes to video support. Once I'm there, I can start looking at doing some fancier stuff...

Ori 

On Sun, Dec 14, 2014 at 2:59 AM, Christophe De Wagter <address@hidden> wrote:
What you describe is not in the repo at the moment but has been done before: a simple module.xml with the following start and stop functions: 

void gst_video_start(void)
{
  // Start GST plugiun
  // Normal video 320
  system("/opt/arm/gst/bin/gst-launch v4l2src device=/dev/video1 ! videorate ! 'video/x-raw-yuv,framerate=15/1' ! videoscale ! video/x-raw-yuv, width=320, height=240 ! dspmp4venc ! rtpmp4vpay config-interval=2 ! udpsink host=192.168.1.255 port=5000 &");
}

void gst_video_stop(void)
{
  // Stop GST-Plugin
  system("kill -9 `pidof gst-launch-0.10` &");
}

and raw:

      <raw>
upload_extra:
@echo "====================================VISION START/INSTALL======================================"
ardron2.py startvision

    </raw>



 we have 

-Christophe 

On Sun, Dec 7, 2014 at 4:53 PM, Ori Pessach <address@hidden> wrote:
Also, I couldn't figure out from the documentation how to load any modules other than the ardrone2_vision mjpeg module... It sounds like there's a way to include the ardrone2_gstreamer module in a paparazzi airframe file so that it sets everything up and starts a gstreamer pipeline when you press upload, but I couldn't figure out how to do it.

Any tips?


On Sun, Dec 7, 2014 at 7:54 AM, Christophe De Wagter <address@hidden> wrote:
ardrone2_vision contains mainly computer vision code and is work in progress.

ardrone2_gstreamer is quite ready and has what you ask for: there are a few examples of gstreamer chains and the upload button has all the needed code to upload images to the dsp and start it. 

-Christophe 

On Sat, Dec 6, 2014 at 10:58 PM, Ori Pessach <address@hidden> wrote:
Hi,

I'm trying to get the vision framework to work, but something is amiss. I can get the mjpeg streaming module to work (by including it in the airframe file) but the image quality is quite poor, and there's a lot of latency when viewed with VLC. I'm guessing it's streamed over a TCP connection, which can do that. 

What I would like to do is try to run the H.264 encoder on the DSP using gstreamer, and save the video to a USB thumb drive on the vehicle, at the very least. This should give the drone similar functionality to what you would get with the Parrot firmware. 

As a first step, I'm trying to just run a pipeline to encode MPEG4 on the drone using gst-launch, (as described here: http://lists.gnu.org/archive/html/paparazzi-devel/2013-11/msg00184.html

This kinda works, after futzing with environment variables on the command line and stopping paparazzi (since with the MJPEG module, it grabs /dev/video1 and we can't have that now, can we?)

So, seeing that I can get gst-launch to work on the drone, I should be able to write a paparazzi module using gstreamer to run the same pipeline and use it to replace the mjpeg module. Right?

What I'm wondering about, though, is why the H.264 DSP based encoder doesn't get installed along with the MPEG-4 encoder. Is this intentional? Is it an oversight? Can I just copy it over, launch it in the same way that ardrone2.py launces the MPEG-4 encoder and expect it to work? Am I missing something?

I would love it if someone who's familiar with the code and how it's supposed to work would comment on that.

Thanks,

Ori Pessach




_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel



_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel



_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel


_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel



_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel


_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel



_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel


_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel



reply via email to

[Prev in Thread] Current Thread [Next in Thread]