The button was showing up no matter what page the user was on. I had to resort to turning it off by editing the theme as you suggested.

I looked for a way to disallow user submitted pages completely but I can’t figure it out. And I don’t see anywhere to do that in the “individual page settings.”