fusion i3d models

[bouha0]

i need help to design or config model to fusion two i3d models , the fusion will flexible from the layer 1 and so on .

i trained i"d models on rgb , optical flow separately i wan to fuse them at different stage of resnet50 i3d

[cir7]

This is a little complicated, I could give you some hints about how to implement it.
It relates to the following components, pipeline, dataset, and model.

For pipeline part, you need to build an input tensor concatenate rgb+flow. While a normal pipeline supports single modality input (rgb or flow), you could create a new dataset using two pipelines to process rgb and flow separately. Another solution is to create a series of transforms that could process rgb+flow, you could refer to the rgb+pose c3d, which takes rgb and skeleton as input.

The model part is relatively simple, a composite model that consists of 2 i3d backbones is ok, you could define the fusion actions as you want in the forward function. RGBPoseConv3D can also be a good example.

What's more, I think late fusion is a much easier way, lol

[bouha0]

i get this :
TypeError: class MMRecognizer3D in mmaction/models/recognizers/recognizer3d_mm.py: tuple indices must be integers or slices, not str

i do some change on rgbposeconv3D and head

[bouha0]

thanks , will send you my code please check it

…

On Tue, 27 Jun 2023 at 10:27, cir7 ***@***.***> wrote: This is a little complicated, I could give you some hints about how to implement it. It relates to the following components, pipeline, dataset, and model. For pipeline part, you need to build an input tensor concatenate rgb+flow. While a normal pipeline supports single modality input (rgb or flow), you could create a new dataset using two pipelines to process rgb and flow separately. Another solution is to create a series of transforms that could process rgb+flow, you could refer to the rgb+pose c3d <https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py>, which takes rgb and skeleton as input. The model part is relatively simple, a composite model that consists of 2 i3d backbones is ok, you could define the fusion actions as you want in the forward function. RGBPoseConv3D <https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py> can also be a good example. What's more, I think late fusion is a much easier way, lol — Reply to this email directly, view it on GitHub <#2546 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFLV2LZEH6KCIHGDSYFDRXLXNKRPVANCNFSM6AAAAAAZKJ6TG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *Bouhafs Abdelkader * *0668131628* *In Salah *

[bouha0]

Hi I coded the mmresnet3d to train , build dataset and alos model i create a new recognizer TwoStreamRecognizer registered as Model under Recognizers On Wed, 28 Jun 2023 at 14:54, Maths & Electronics Tutos < ***@***.***> wrote:

…

thanks , will send you my code please check it On Tue, 27 Jun 2023 at 10:27, cir7 ***@***.***> wrote: > This is a little complicated, I could give you some hints about how to > implement it. > It relates to the following components, pipeline, dataset, and model. > > For pipeline part, you need to build an input tensor concatenate > rgb+flow. While a normal pipeline supports single modality input (rgb or > flow), you could create a new dataset using two pipelines to process rgb > and flow separately. Another solution is to create a series of transforms > that could process rgb+flow, you could refer to the rgb+pose c3d > <https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py>, > which takes rgb and skeleton as input. > > The model part is relatively simple, a composite model that consists of 2 > i3d backbones is ok, you could define the fusion actions as you want in the > forward function. RGBPoseConv3D > <https://github.com/open-mmlab/mmaction2/blob/main/configs/skeleton/posec3d/rgbpose_conv3d/rgbpose_conv3d.py> > can also be a good example. > > What's more, I think late fusion is a much easier way, lol > > — > Reply to this email directly, view it on GitHub > <#2546 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFLV2LZEH6KCIHGDSYFDRXLXNKRPVANCNFSM6AAAAAAZKJ6TG4> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- *Bouhafs Abdelkader * *0668131628* *In Salah *

-- *Bouhafs Abdelkader * *0668131628* *In Salah *