Data

[FightingFighting]

Hi, Thanks for your amazing work! I wonder when will you release your code and data? Thanks!

[phellonchen]

We release the code and the data for training the image/speech/video interface. The voice and speech features will be uploaded soon. Please see README_DATA.md

[FightingFighting]

@phellonchen Thank you for quick reply.

it is seem that these is a problem that the dataset Visual Genome is not right.

[FightingFighting]

@phellonchen and can you double check flickr30k dataset? it seems that the image link gets something wrong.

[phellonchen]

@FightingFighting Images of flickr30k dataset have been saved in the json file. Its format is as follows:

[
   {
       "image": image_base64
       "caption": [
             "两个年轻小伙子的头发浓密的看着他们的手在院子里闲逛。",
            "两只年轻的，白色的男性在附近的许多灌木丛。",
            "绿色衬衫的男人站在一个院子里。",
            "一个穿蓝色衬衫的人站在花园里。",
            "朋友们一起享受时光。
        ]
   }
]

you can use the code to use the base64 image:

from io import BytesIO
from PIL import Image
import base64
image = Image.open(BytesIO(base64.urlsafe_b64decode(ann["image"]))).convert("RGB")

[phellonchen]

@phellonchen Thank you for quick reply.

it is seem that these is a problem that the dataset Visual Genome is not right.

It looks like there is an issue with the official website. You can try downloading this dataset using HuggingFace instead.