Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add motion vector extraction from side data #33

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions avutil/avutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ package avutil
//#include <libavutil/parseutils.h>
//#include <libavutil/common.h>
//#include <libavutil/eval.h>
//#include <libavutil/motion_vector.h>
//
//#ifdef AV_LOG_TRACE
//#define GO_AV_LOG_TRACE AV_LOG_TRACE
Expand Down Expand Up @@ -42,6 +43,16 @@ package avutil
// return 0;
//}
//
//static const AVMotionVector *go_motion_vector(const AVFrameSideData *sd, int num) {
// const AVMotionVector *mvs = (const AVMotionVector *)sd->data;
// return &mvs[num];
//}
//
//static int go_motion_vector_size(const AVFrameSideData *sd) {
// const AVMotionVector *mvs = (const AVMotionVector*)sd->data;
// return sd->size / sizeof(*mvs);
//}
//
//static const int go_av_errno_to_error(int e)
//{
// return AVERROR(e);
Expand Down Expand Up @@ -105,6 +116,24 @@ const (
PictureTypeBI PictureType = C.AV_PICTURE_TYPE_BI
)

type FrameSideDataType C.enum_AVFrameSideDataType

const (
FrameDataPanScan FrameSideDataType = C.AV_FRAME_DATA_PANSCAN
FrameDataA53CC FrameSideDataType = C.AV_FRAME_DATA_A53_CC
FrameDataStereo3D FrameSideDataType = C.AV_FRAME_DATA_STEREO3D
FrameDataMatrixEncoding FrameSideDataType = C.AV_FRAME_DATA_MATRIXENCODING
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should use _ too in MATRIX_ENCODING

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean we use FrameDataMATRIX_ENCODING instead of FrameDataMatrixEncoding?

Should I change all these types using _?

Thanks.

FrameDataDownMixInfo FrameSideDataType = C.AV_FRAME_DATA_DOWNMIX_INFO
FrameDataReplayGain FrameSideDataType = C.AV_FRAME_DATA_REPLAYGAIN
FrameDataDisplayMatrix FrameSideDataType = C.AV_FRAME_DATA_DISPLAYMATRIX
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DISPLAY_MATRIX

FrameDataAFD FrameSideDataType = C.AV_FRAME_DATA_AFD
FrameDataMotionVectors FrameSideDataType = C.AV_FRAME_DATA_MOTION_VECTORS
FrameDataSkipSamples FrameSideDataType = C.AV_FRAME_DATA_SKIP_SAMPLES
FrameDataAudioServiceType FrameSideDataType = C.AV_FRAME_DATA_AUDIO_SERVICE_TYPE
FrameDataMasteringDisplayMetadata FrameSideDataType = C.AV_FRAME_DATA_MASTERING_DISPLAY_METADATA
FrameDataGopTimecode FrameSideDataType = C.AV_FRAME_DATA_GOP_TIMECODE
)

type ChromaLocation C.enum_AVChromaLocation

const (
Expand Down Expand Up @@ -980,6 +1009,110 @@ func (f *Frame) PacketDuration() int64 {
return int64(C.av_frame_get_pkt_duration(f.CAVFrame))
}

func (f *Frame) SideData(dataType FrameSideDataType) *FrameSideData {
cSideData := C.av_frame_get_side_data(f.CAVFrame, (C.enum_AVFrameSideDataType)(dataType))
if cSideData == nil {
return nil
}

return NewFrameSideDataFromC(unsafe.Pointer(cSideData))
}

type FrameSideData struct {
CFrameSideData *C.AVFrameSideData
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I am following this pattern, can you follow it too?

func NewFrameSideDataFromC(cFrameSideData unsafe.Pointer) * FrameSideData {
    return &FrameSideData{CFrameSideData: (*C.AVFrameSideData)(cFrameSideData)}
}

and then call it inside Frame.SideData() ?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add

func (sd *FrameSideData) Type() FrameSideDataType {
  return FrameSideDataType(sd.type)
}

func NewFrameSideDataFromC(cFrameSideData unsafe.Pointer) *FrameSideData {
return &FrameSideData{
CFrameSideData: (*C.AVFrameSideData)(cFrameSideData),
}
}

func (sd *FrameSideData) Size() int {
return int(sd.CFrameSideData.size)
}

func (sd *FrameSideData) Type() FrameSideDataType {
return FrameSideDataType(sd.CFrameSideData._type)
}

func (sd *FrameSideData) MotionVector(n int) *MotionVector {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you didn't follow my advice for MotionVectors() []*MotionVector.
Also, you're not using go_motion_vector_size anymore.
How will you discover the number of motion vectors in go?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about this

mv := C.go_motion_vector(sd.CFrameSideData, C.int(n))
return NewMotionVectorFromC(unsafe.Pointer(&mv))
}

func (sd *FrameSideData) Metadata() *Dictionary {
if sd.CFrameSideData.metadata == nil {
return nil
}

return NewDictionaryFromC(unsafe.Pointer(&sd.CFrameSideData.metadata))
}

func (sd *FrameSideData) Free() {
if sd == nil {
return
}

C.av_buffer_unref(unsafe.Pointer(&sd.CFrameSideData.buf))
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't these managed pointers. Do we to provide this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use buf directly here, but we need to free it.

I will dig more about the buf and av_buffer_unref usage.

Thanks.

sd.Metadata().Free()
C.av_freep(unsafe.Pointer(sd.CFrameSideData))
}

type MotionVector struct {
CMotionVector *C.AVMotionVector
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you do the same here for NewMotionVectorFromC ?

func NewMotionVectorFromC(mv unsafe.Pointer) *MotionVector {
return &MotionVector{
CMotionVector: (*C.AVMotionVector)(mv),
}
}

func (mv *MotionVector) Source() int32 {
return int32(mv.CMotionVector.source)
}

func (mv *MotionVector) Width() uint8 {
return uint8(mv.CMotionVector.w)
}

func (mv *MotionVector) Height() uint8 {
return uint8(mv.CMotionVector.h)
}

func (mv *MotionVector) SrcX() int16 {
return int16(mv.CMotionVector.src_x)
}

func (mv *MotionVector) SrcY() int16 {
return int16(mv.CMotionVector.src_y)
}

func (mv *MotionVector) DstX() int16 {
return int16(mv.CMotionVector.dst_x)
}

func (mv *MotionVector) DstY() int16 {
return int16(mv.CMotionVector.dst_y)
}

func (mv *MotionVector) Flags() uint64 {
return uint64(mv.CMotionVector.flags)
}

func (mv *MotionVector) MotionX() int32 {
return int32(mv.CMotionVector.motion_x)
}

func (mv *MotionVector) MotionY() int32 {
return int32(mv.CMotionVector.motion_y)
}

func (mv *MotionVector) MotionScale() uint16 {
return uint16(mv.CMotionVector.motion_scale)
}

type OptionAccessor struct {
obj unsafe.Pointer
fake bool
Expand Down
10 changes: 10 additions & 0 deletions avutil/avutil_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -845,3 +845,13 @@ func TestPixelFormatDescriptor_ComponentCount(t *testing.T) {
t.Fatalf("[TestPixelFormatDescriptor_ComponentCount] count=%d, NG expected=%d", count, 3)
}
}

func TestMotionVectorFromFrameSideData(t *testing.T) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make a separate function TestFrameSideData where you call all its funcs?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problems. I will try your test videos and random generated videos. 😄

frame, _ := NewFrame()
defer frame.Free()

sideData := frame.SideData(FrameDataMotionVectors)
if sideData != nil {
t.Fatalf("[TestMotionVectorFromFrameSideData] sidedata=%v, NG expected=nil", sideData)
}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you call all MotionVector's funcs?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problems.

}