Add filter to oembed only in front end

You can utilize the is_admin() function. https://codex.wordpress.org/Function_Reference/is_admin

This should work, however I don’t like running it when we don’t need it:

add_filter('embed_oembed_html', 'my_embed_oembed_html', 99, 4);
function my_embed_oembed_html($html, $url, $attr, $post_id) {
  if ( ! is_admin() ) {
    return '<div class="embed-responsive embed-responsive-16by9">' . $html . '</div>';
  }
}

This may work, test it first.

if ( ! is_admin() ) {
  add_filter('embed_oembed_html', 'my_embed_oembed_html', 99, 4);
}
function my_embed_oembed_html($html, $url, $attr, $post_id) {
  return '<div class="embed-responsive embed-responsive-16by9">' . $html . '</div>';
}