is_page_template in functions.php not working.

I have this in functions.php:

if( is_page_template( 'my-templates/contact.php' ) ) add_action( 'wp_enqueue_scripts', 'contact_script_enqueuer' );

When I load my Contact page, which uses the contact template, the scripts do not load. If I get rid of the condition, they load fine.

Any ideas?