If any of your posts begin with a symbol (i.e. not A-Z nor 0-9) then a new letter will show labelled #, which would conflict with your numbers letter. However, you can do this by hooking the the-a-z-letter-title filter. You could add something like this to your theme’s (or child theme’s) functions.php file:
add_filter( 'the-a-z-letter-title', 'replace_a_z_numbers_title' );
function replace_a_z_numbers_title( $letter ) {
// The numbers letter is '0' when the numbers are grouped into one letter.
if ( '0' === $letter ) {
return '#';
}
// We're not working on the numbers group so return the original letter unchanged
return $letter;
}