How to make video and audio timecodes start from zero?

Question

This question will probably require the knowledge of the matroska format. When I mux video and audio together, audio packets start with timecode 0, and video starts with timecode 7. Should a 30fps video start with timecode 0, or with timecode 33 (for time 0.033, timecodes are in milliseconds)? Can I make it so that both audio and video start with timecode 0?

You don't have to read below, the rest is just additional info.

---

I tried to add filters [v]setpts=N/(30*TB)[v];[a]asetpts=N/SR/TB[a] as the two last filters, but it didn't change anything.

Here is the full command I'm using. It's just a two pass vp8 encoding for video and one pass opus encoding for audio. It also chops a video in a few 3 second long pieces, and joins them together.

ffmpeg -threads 1 -i 480P_600K_71149981.mp4 -force_key_frames 00:00:03.000 -filter_complex [0:v]scale=320x180:force_original_aspect_ratio=decrease,fps=30[vid];[vid]split[vid][vid_copy];[vid_copy]trim=start=0:duration=3,setpts=PTS-STARTPTS[c0v];[0:a]atrim=start=0:duration=3,asetpts=PTS-STARTPTS[c0a];[vid]split[vid][vid_copy];[vid_copy]trim=start=90:duration=3,setpts=PTS-STARTPTS[c1v];[0:a]atrim=start=90:duration=3,asetpts=PTS-STARTPTS[c1a];[vid]split[vid][vid_copy];[vid_copy]trim=start=180:duration=3,setpts=PTS-STARTPTS[c2v];[0:a]atrim=start=180:duration=3,asetpts=PTS-STARTPTS[c2a];[vid]trim=start=270:duration=3,setpts=PTS-STARTPTS[c3v];[0:a]atrim=start=270:duration=3,asetpts=PTS-STARTPTS[c3a];[c0v][c0a][c1v][c1a][c2v][c2a][c3v][c3a]concat=n=4:v=1:a=1[v][a];[v]setpts=N/(30*TB)[v];[a]asetpts=N/SR/TB[a] -map [v] -map [a] -c:v vp8 -b:v 200k -crf 54 -profile:v 1 -an -pass 1 -passlogfile jump_passlogfile -f null NUL


ffmpeg -threads 1 -i 480P_600K_71149981.mp4 -force_key_frames 00:00:03.000 -filter_complex [0:v]scale=320x180:force_original_aspect_ratio=decrease,fps=30[vid];[vid]split[vid][vid_copy];[vid_copy]trim=start=0:duration=3,setpts=PTS-STARTPTS[c0v];[0:a]atrim=start=0:duration=3,asetpts=PTS-STARTPTS[c0a];[vid]split[vid][vid_copy];[vid_copy]trim=start=90:duration=3,setpts=PTS-STARTPTS[c1v];[0:a]atrim=start=90:duration=3,asetpts=PTS-STARTPTS[c1a];[vid]split[vid][vid_copy];[vid_copy]trim=start=180:duration=3,setpts=PTS-STARTPTS[c2v];[0:a]atrim=start=180:duration=3,asetpts=PTS-STARTPTS[c2a];[vid]trim=start=270:duration=3,setpts=PTS-STARTPTS[c3v];[0:a]atrim=start=270:duration=3,asetpts=PTS-STARTPTS[c3a];[c0v][c0a][c1v][c1a][c2v][c2a][c3v][c3a]concat=n=4:v=1:a=1[v][a];[v]setpts=N/(30*TB)[v];[a]asetpts=N/SR/TB[a] -map [v] -map [a] -c:v vp8 -b:v 200k -crf 54 -profile:v 1 -c:a libopus -b:a 32k -vbr on -compression_level 7 -ac 1 -ar 48000 -pass 2 -passlogfile jump_passlogfile -f webm -reserve_index_space 512 480P_600K_71149981_vthumb.webm

Here is the 480P_600K_71149981.mp4 file I'm testing this on.

And here is the result, 480P_600K_71149981_vthumb.webm and the text dump of it created with webm_parser_demo.exe from libwebm (slightly modified to display hex instead of decimal). The interesting parts of the text dump are, search for "timecode:" and "Cluster".

  Cluster                            header: [309, 311)  body: [311, 3487)
    Timecode: 0
    SimpleBlock                      header: [31a, 31c)  body: [31c, 323)
      track number: 2
      frames: 1
      timecode: 0
      lacing: 0 (none)
      flags: visible, key frame
      frame byte range: [320, 323)
    SimpleBlock                      header: [323, 325)  body: [325, 395)
      track number: 1
      frames: 1
      timecode: 7
      lacing: 0 (none)
      flags: visible, key frame
      frame byte range: [329, 395)
    SimpleBlock                      header: [395, 397)  body: [397, 39e)
      track number: 2
      frames: 1
      timecode: 15
      lacing: 0 (none)
      flags: visible, key frame
      frame byte range: [39b, 39e)
    SimpleBlock                      header: [39e, 3a0)  body: [3a0, 3bb)
      track number: 1
      frames: 1
      timecode: 28
      lacing: 0 (none)
      flags: visible
      frame byte range: [3a4, 3bb)
    SimpleBlock                      header: [3bb, 3bd)  body: [3bd, 3c4)
      track number: 2
      frames: 1
      timecode: 29
      lacing: 0 (none)
      flags: visible, key frame
      frame byte range: [3c1, 3c4)

Source and destination video have different fps, this is probably important. There are setpts=PTS-STARTPTS filters, but they oddly don't affect the starting Presentation TimeStamp.

This is important to me to make sure that the -force_key_frames creates keyframe at the correct time, and later I will use timecodes to leave only 1 cue in the file. I'm just trying to polish it, trying to learn video formats, while I have a chance. I will make a video player for those later, need to make sure those files are as simple to read and render as possible.

If there is only video, or only audio, timecodes start properly from 0. I tested it with this command:

ffmpeg -i 480P_600K_71149981.mp4 -t 3 -an -reserve_index_space 512 out.webm

Also I wonder if it's possible to make video frames go before audio frames, it seems like a more natural way of storing them, though I'm not sure about anything. It will probably solve itself once the timestamps problem is solved.

If solution requires changing anything in ffmpeg and recompiling, I'm fine with it. Just need to find what to change first.

Though, I wonder if I should actually keep audio before video, and use -audio_preload. Because to seek opus audio, I have to replay SeekPreRoll milliseconds anyway (80ms in my case). Not like I care about that in this case, I don't intend to ever seek inside a 12 seconds video, I only need to quickly grab the keyframe from second 3 of a video for the thumbnail. Yes, no -audio_preload here. Also, there are more of some confusing info about opus muxing in matroska here, probably useless in this case: https://wiki.xiph.org/MatroskaOpus .

Wait, I think I figured out why video is delayed by 7 milliseconds. Because in opus, you have to play CodecDelay (6.5ms here) of audio and just discard it, before any audio can be heard, just to initialize the decoder. Hm, now I wonder if I can set this discarded audio to a negative timestamp. https://www.matroska.org/technical/specs/index.html#simpleblock_structure the timecode here is a signed int16, so probably I can. But how to tell ffmpeg this...

---

Edit1: here is the output of ffprobe:

ffprobe 480P_600K_71149981_vthumb.webm -show_packets -select_streams v -read_intervals %+#5 -v 0

-

[PACKET]
codec_type=video
stream_index=0
pts=7
pts_time=0.007000
dts=7
dts_time=0.007000
duration=33
duration_time=0.033000
convergence_duration=N/A
convergence_duration_time=N/A
size=108
pos=897
flags=K_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=40
pts_time=0.040000
dts=40
dts_time=0.040000
duration=33
duration_time=0.033000
convergence_duration=N/A
convergence_duration_time=N/A
size=23
pos=1020
flags=__
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=74
pts_time=0.074000
dts=74
dts_time=0.074000
duration=33
duration_time=0.033000
convergence_duration=N/A
convergence_duration_time=N/A
size=23
pos=1067
flags=__
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=107
pts_time=0.107000
dts=107
dts_time=0.107000
duration=33
duration_time=0.033000
convergence_duration=N/A
convergence_duration_time=N/A
size=23
pos=1114
flags=__
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=140
pts_time=0.140000
dts=140
dts_time=0.140000
duration=33
duration_time=0.033000
convergence_duration=N/A
convergence_duration_time=N/A
size=23
pos=1152
flags=__
[/PACKET]

--

ffprobe 480P_600K_71149981_vthumb.webm -show_packets -select_streams a -read_intervals %+#5 -v 0

-

[PACKET]
codec_type=audio
stream_index=1
pts=-7
pts_time=-0.007000
dts=-7
dts_time=-0.007000
duration=20
duration_time=0.020000
convergence_duration=N/A
convergence_duration_time=N/A
size=3
pos=888
flags=K_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=14
pts_time=0.014000
dts=14
dts_time=0.014000
duration=20
duration_time=0.020000
convergence_duration=N/A
convergence_duration_time=N/A
size=3
pos=1011
flags=K_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=34
pts_time=0.034000
dts=34
dts_time=0.034000
duration=20
duration_time=0.020000
convergence_duration=N/A
convergence_duration_time=N/A
size=3
pos=1049
flags=K_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=54
pts_time=0.054000
dts=54
dts_time=0.054000
duration=20
duration_time=0.020000
convergence_duration=N/A
convergence_duration_time=N/A
size=3
pos=1058
flags=K_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=74
pts_time=0.074000
dts=74
dts_time=0.074000
duration=20
duration_time=0.020000
convergence_duration=N/A
convergence_duration_time=N/A
size=3
pos=1096
flags=K_
[/PACKET]

Really weird how video has the first timestamp 7, and audio has first timestamp -7.

In the matroska (checked with webm_parser_demo.exe), video has timestamp 7, and audio has timestamp 0.

And my goal is to make first video timestamp 0, and first audio timestamp -7 (all the CodecDelay data).

---

Edit2: Studied a few webm with vorbis audio, they also don't start both at 0. In them, audio starts at 0, and video at 3. Don't know what's up with that, is it also CodecDelay?