upload_mimes filter not working correctly

So it appears that the upload_mimes filter is not working as expected. After much research and compared my code to multiple examples, I cannot figure out why it is not working. So in my code, I set what file types to accept and unset ALL AUDIO & VIDEO file types. But I can still upload our mp4 vidoes.

add_filter( 'upload_mimes', 'my_mime_types', 1, 1 );
if ( ! function_exists( 'my_mime_types' ) ) {
    function my_mime_types( $mime_types  ) {
        $mime_types['jpg'] = 'image/jpeg';
        $mime_types['png'] = 'image/png';
        $mime_types['pdf'] = 'application/pdf';
        $mime_types['doc'] = 'application/msword';
        $mime_types['docx'] = 'application/vnd.openxmlformats-officedocument.wordprocessingml.document';
        $mime_types['kmz'] = 'application/vnd.google-earth.kml+xml';

        unset($mime_types['mp3']);
        unset($mime_types['m4a']);
        unset($mime_types['wav']);
        unset($mime_types['ogg']);
        unset($mime_types['mp4']);
        unset($mime_types['mov']);
        unset($mime_types['wmv']);
        unset($mime_types['avi']);
        unset($mime_types['mpg']);
        unset($mime_types['ogv']);
        unset($mime_types['3gp']);
        unset($mime_types['3g2']);
        return $mime_types;
    }
}

Thanks!