modify passed by reference vars in filter hooks

I’m trying to modify the widget arguments (like $args['before_widget']) before it is passed to widget() function, wordpress has widget_display_callback filter hook which passes the $instance, $widget and $args to my function, but I can’t modify the $args variable. Now, if I try to pass the $args by reference, I get “expected to be a reference, value given” error. Is there a way around this?