Otherwise they'd be silently ignored if received by the filtering thread before the filtergraph can be initialized, which would make the output dependent on the order in which frames from different inputs arrive.