Is it possible that show depth and color image simultaneously

[Wesley-Lin]

I follow NiViewer.Net sample to develop my code, but I find out I can't show both depth and color in the same time. I used two Devices class to show, but not work, please give some advices
Thanks in advance.

[falahati]

Hey,

You should use only one device class in your code and start the depth stream before starting the color stream. As far as I remember, you also can't increase the resolution of the color stream above a certain threshold when both streams are active because of USB's bandwidth limitations.
That being said, there is nothing special or tricky about having both streams active at the same time.

If you couldn't solve the problem after following these suggestions, put important parts of your code so I can at least act as another eye to help you in the process.

Also, it may worth to mention that some people had a problem with video stream freezing after a while when both streams are active. As described here #16; yet I failed to reproduce or fix it, yet.

[Wesley-Lin]

Hi Falahati,

I use one device class and both are the same resolution (640 x 480 ), but still not work. Every time I start depth stream then start color stream, the depth stream will be stopped automatically.
Following is my part of code (C# application).

        private void Form1_Load(object sender, EventArgs e)
        {
            color_bitmap = new Bitmap(640, 480);
            depth_bitmap = new Bitmap(640, 480);
            this.HandleError(OpenNI.Initialize());  // HandleError() function is like your sample
            OpenNI.OnDeviceConnected += this.OpenNiOnDeviceConnectionStateChanged;  // OpenNiOnDeviceConnectionStateChanged() function is also like yours.
            OpenNI.OnDeviceDisconnected += this.OpenNiOnDeviceConnectionStateChanged;
            this.UpdateDevicesList();  // function is like yours too.

            currentDevice = devices[0].OpenDevice();
        }

        private void Btn_RGBStart_Click(object sender, EventArgs e)
        {
            if (this.color_stream != null)
            {
                VideoMode colorMode = new VideoMode
                {
                    Resolution = new Size(640, 480),
                    Fps = 30,
                    DataPixelFormat = VideoMode.PixelFormat.Rgb888
                };
                this.color_stream.VideoMode = colorMode;

                if (this.color_stream.Start() == OpenNI.Status.Ok)
                {
                    this.color_stream.OnNewFrame += RgbOnNewFrame;
                }
                else
                {
                    MessageBox.Show(@"Failed to start RGB stream.");
                }
                
            }
        }

        private void RgbOnNewFrame(VideoStream vs)
        {
            if (vs.IsValid && vs.IsFrameAvailable())
            {
                using (VideoFrameRef frame = vs.ReadFrame())
                {
                    if (frame.IsValid)
                    {
                        VideoFrameRef.CopyBitmapOptions options = VideoFrameRef.CopyBitmapOptions.Force24BitRgb;
                        lock (this.color_bitmap)
                        {
                            try
                            {
                                frame.UpdateBitmap(color_bitmap, options);
                            }
                            catch (Exception)
                            {

                                color_bitmap = frame.ToBitmap(options);
                            }

                        }
                        this.BeginInvoke(new MethodInvoker(delegate()
                        {
                            if (color_bitmap == null)
                            {
                                return;
                            }
                            lock (color_bitmap)
                            {
                                PB_RGB.Image = color_bitmap;
                                PB_RGB.Refresh();
                            }
                        }));
                        
                    }
                }
            }
        }

        private void Btn_DepthStart_Click(object sender, EventArgs e)
        {
            if (this.depth_stream != null)
            {

                VideoMode depthrMode = new VideoMode
                {
                    Resolution = new Size(640, 480),
                    Fps = 30,
                    DataPixelFormat = VideoMode.PixelFormat.Depth1Mm
                };
                this.depth_stream.VideoMode = depthrMode;

                if (this.depth_stream.Start() == OpenNI.Status.Ok)
                {
                    this.depth_stream.OnNewFrame += DepthOnNewFrame;
                }
                else
                {
                    MessageBox.Show(@"Failed to start Depth stream.");
                }
                
            }
        }

        private void DepthOnNewFrame(VideoStream vs)
        {
            Console.WriteLine("Depth New Frame");
            if (vs.IsValid && vs.IsFrameAvailable())
            {
                using (VideoFrameRef frame = vs.ReadFrame())
                {
                    if (frame.IsValid)
                    {
                        VideoFrameRef.CopyBitmapOptions options = VideoFrameRef.CopyBitmapOptions.Force24BitRgb
                                                                  | VideoFrameRef.CopyBitmapOptions.DepthFillShadow;
                        lock (this.depth_bitmap)
                        {
                            try
                            {
                                frame.UpdateBitmap(depth_bitmap, options);
                                Console.WriteLine("Depth Frame Get");
                                Console.WriteLine("=================");
                            }
                            catch (Exception)
                            {

                                depth_bitmap = frame.ToBitmap(options);
                            }
                        }
                        this.BeginInvoke(new MethodInvoker(delegate()
                        {
                            if (depth_bitmap == null)
                            {
                                return;
                            }
                            lock (depth_bitmap)
                            {
                                PB_Depth.Image = depth_bitmap;
                                PB_Depth.Refresh();
                            }
                        }));
                       
                    }
                }
            }
        }

please review my source codes, any pointers to this issue will be highly appreciated
Thank you.

[falahati]

Following worked perfectly on my system with Asus Xtion, it is almost the same code you provided and uses the latest version of the NiWrapper.

        private Bitmap _colorBitmap;
        private VideoStream _colorStream;
        private Device _currentDevice;
        private Bitmap _depthBitmap;
        private VideoStream _depthStream;

        public Form1()
        {
            InitializeComponent();
        }

        private void Btn_DepthStart_Click(object sender, EventArgs e)
        {
            if (_depthStream == null)
            {
                _depthStream = _currentDevice.CreateVideoStream(Device.SensorType.Depth);
                var mode = new VideoMode
                {
                    Resolution = new Size(640, 480),
                    Fps = 30,
                    DataPixelFormat = VideoMode.PixelFormat.Depth1Mm
                };
                _depthStream.VideoMode = mode;

                if (_depthStream.Start() == OpenNI.Status.Ok)
                {
                    _depthStream.OnNewFrame += DepthOnNewFrame;
                    Btn_DepthStart.Enabled = false;
                }
                else
                {
                    MessageBox.Show(@"Failed to start Depth stream.");
                }
            }
        }


        private void Btn_RGBStart_Click(object sender, EventArgs e)
        {
            if (_colorStream == null)
            {
                _colorStream = _currentDevice.CreateVideoStream(Device.SensorType.Color);
                var colorMode = new VideoMode
                {
                    Resolution = new Size(640, 480),
                    Fps = 30,
                    DataPixelFormat = VideoMode.PixelFormat.Rgb888
                };
                _colorStream.VideoMode = colorMode;

                if (_colorStream.Start() == OpenNI.Status.Ok)
                {
                    _colorStream.OnNewFrame += RgbOnNewFrame;
                    Btn_RGBStart.Enabled = false;
                }
                else
                {
                    MessageBox.Show(@"Failed to start RGB stream.");
                }
            }
        }

        private void DepthOnNewFrame(VideoStream vs)
        {
            if (vs.IsValid && vs.IsFrameAvailable())
            {
                using (var frame = vs.ReadFrame())
                {
                    if (frame.IsValid)
                    {
                        var options = VideoFrameRef.CopyBitmapOptions.Force24BitRgb |
                                      VideoFrameRef.CopyBitmapOptions.DepthFillShadow;

                        lock (this)
                        {
                            try
                            {
                                frame.UpdateBitmap(_depthBitmap, options);
                            }
                            catch (Exception)
                            {
                                _depthBitmap = frame.ToBitmap(options);
                            }
                        }

                        BeginInvoke(new MethodInvoker(delegate
                        {
                            if (_depthBitmap == null)
                            {
                                return;
                            }

                            lock (this)
                            {
                                PB_Depth.Image = _depthBitmap;
                                PB_Depth.Refresh();
                            }
                        }));
                    }
                }
            }
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            HandleError(OpenNI.Initialize());
            _currentDevice = Device.Open(null);
        }


        private void HandleError(OpenNI.Status status)
        {
            if (status == OpenNI.Status.Ok)
            {
                return;
            }

            MessageBox.Show(
                string.Format(@"Error: {0} - {1}", status, OpenNI.LastError),
                @"Error",
                MessageBoxButtons.OK,
                MessageBoxIcon.Asterisk);
        }

        private void RgbOnNewFrame(VideoStream vs)
        {
            if (vs.IsValid && vs.IsFrameAvailable())
            {
                using (var frame = vs.ReadFrame())
                {
                    if (frame.IsValid)
                    {
                        var options = VideoFrameRef.CopyBitmapOptions.Force24BitRgb;

                        lock (this)
                        {
                            try
                            {
                                frame.UpdateBitmap(_colorBitmap, options);
                            }
                            catch (Exception)
                            {
                                _colorBitmap = frame.ToBitmap(options);
                            }
                        }

                        BeginInvoke(new MethodInvoker(delegate
                        {
                            if (_colorBitmap == null)
                            {
                                return;
                            }

                            lock (this)
                            {
                                PB_RGB.Image = _colorBitmap;
                                PB_RGB.Refresh();
                            }
                        }));
                    }
                }
            }
        }

[falahati]

If you still have a problem with this, your issue is probably same as #16. If your device is Kinect, you might be limited to 320x240; I am not totally sure tho.