nutdec: check maxpos in read_sm_data before returning success

Otherwise sm_size can be larger than size, which results in a negative
packet size.

Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 6b9fdf7f4f07926557048070cc2af3cfd0e3fe50)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>

libavformat/nutdec.c

@@ -963,6 +963,9 @@ static int read_sm_data(AVFormatContext *s, AVIOContext *bc, AVPacket *pkt, int
         AV_WL32(dst+4, skip_end);
     }
 
+    if (avio_tell(bc) >= maxpos)
+        return AVERROR_INVALIDDATA;
+
     return 0;
 }