You can achieve that by changing users settings. Go to Control Panel » Users » User groups
Pick a user that you don't want to view your forum.
In this example I will pick Users group
Expand +Forum and uncheck Read forum (ability to enter the forum)
Save it and your done.
Do the the same thing for every group that you don't want to see your forum.
