Media library attachments not being access restricted

It seems the “Access restrictions” feature of Groups on the “Edit Media” pages of WordPress doesn’t work on my site as I would expect. The plugin works great for all my other read access restrictions but when I edit a media item specifically and I select certain groups for read access, I can still browse to that media file via the URL in my browser without even being logged into my WordPress site.
I would expect that if I assign read access to that media item, it should be completely protected from being accessed. Otherwise what is the purpose of having that as a screen option on the “Edit Media” page?

https://wordpress.org/plugins/groups/