I want to create a private forum to do research with friends and stuff (no more than 5 people or so).  I want to quickly set it up on my site, a la softaculous or something like that.  I also want to use tapatalk or something similar for mobile use.

What is a receommended software?  My favorite forum software is this one here, just for my personal preference.

Tapatalk compatible forums:

Q: Which forum system does it currently support?
phpBB3, vBulletin, xenForo, IPBoard, Simple Machine, MyBB, bbPress, Kunena and Vanilla are supported. Other forum system might also support Tapatalk with plugins developed by third party developers using Tapatalk API.

(no more than 5 people or so).

with so few people, you ARE a good candidate for some of the more unusual boutique forum software, and smf may be overkill.

and you may benefit from not having to host it yourself.  maybe something like Disqus?