I don’t think you’d want to rely on restricting based on referrer alone for security, as it’s easy to create a http request that forges the referrer. I think Google uses it more to stop someone else serving a web page that would use up your quota.

It would be a nice fit for a public subset of the API tho and would definitely be beneficial!