There is no point in having a per-stream wallclock start time, since they are all computed at the same instant. Keep a per-file start time instead, initialized when the demuxer thread starts.