The encoder may keep a reference to frames that were sent to it, so the caller cannot modify them without checking first.