{
  "title": "Improve Media #3",
  "video_id": "H-xmOFVWlrM",
  "published": "2024-02-20T15:37:12+0000",
  "complexity": "Complex",
  "old_url": "https://workflows-of-ai.com/workflow/improve-media-3",
  "description": "Describe or summarize videos in the Drupal Media module with the AI Interpolator OpenAI video text rule, FFmpeg keyframes and Whisper.",
  "subtitle": "Ask OpenAI to describe a video in Drupal Media, using keyframe grids and a Whisper transcript so it can both see and hear it.",
  "summary": "A new OpenAI video text rule, only in the dev branch at the time, fills a formatted text field on the Video media type from a free prompt. FFmpeg takes a screenshot at each scene cut, the screenshots go into grid images with timestamps, and Whisper adds a timed transcript, so OpenAI gets both picture and sound. Tested on a Tom Scott video, the description covers what is said and what is shown, like aerial views and closeups of towers.",
  "categories": [
    "Drupal AI"
  ],
  "intro_html": "<p>This video is a little bit sloppy, but it's about a development feature - it will be in a release soon.</p><p>I just did some videos about how to set up the AI Interpolator to generate extra media data for the Media module for image, document and audio types.</p><p>This got me thinking about a way to do video as well, and I spent some hours last night coding this. It's currently available in the development version of the module, but hopefully it will also soon be available in the next release.</p><p>What this does is rasterize all the scene cuts/keyframes from the video onto multiple images. Then it also extracts the audio and sends that to OpenAI Whisper to get a transcription.</p><p>Then your prompt is sent together with these two components as context, for it to be able to \"see\" and \"hear\" the video.</p><p>Note that this only works with FFmpeg installed on your server.</p>",
  "intro_added": false,
  "chapters": [
    "0:00 A new rule for video",
    "1:12 Uploading a test video",
    "1:58 Other field types it could fill",
    "3:08 Setting up the video text rule",
    "6:08 How it sees and hears video",
    "8:11 The generated description"
  ],
  "corrections": [
    [
      "AI inator",
      "AI Interpolator"
    ],
    [
      "open AI",
      "OpenAI"
    ],
    [
      "FFM peg",
      "FFmpeg"
    ],
    [
      "FFM Peg",
      "FFmpeg"
    ],
    [
      "have FM Peg",
      "have FFmpeg"
    ],
    [
      "Tom Scot video",
      "Tom Scott video"
    ],
    [
      "AI interpolator",
      "AI Interpolator"
    ],
    [
      "Json",
      "JSON"
    ]
  ],
  "ai_note": "The video and the introduction are Marcus's own, first published on workflows-of-ai.com. AI wrote the subtitle, the summary and the video chapters from the transcript, and corrected misheard names in YouTube's automatic captions."
}