Start using WP-CLI. Nearly every major WordPress project is building add-ons, but even if they weren't, WP-CLI is the hammer in a WP dev's toolbox.
To install WordPress to a current directory:
wp core download
Since you are likely installing on a VPS, that is super fast, and you can then install a bunch of plugins like:
wp plugin install jetpack wordpress-seo disable-google-fonts
That will install each of those from the public repo, but you can check the plugin docs for installing from other sources, such as zip files or a git repo.
As for setting up a site, my workflow is:
VVV (plus VV for new site creation) - full, virtualized WordPress instances (that means they are contained, and can't mess with my system). So I build my sites locally, and adding plugins or themes is normally just copying them into the correct directory.
WP Migrate DB Pro - once the local site is ready and configured, sync this up to production.
wp-content/ remotehost:/path/to/wordpress/wp-content - copies all the theme, plugin and upload files to where they will live online.
I know that looks like a lot, but once it is set up you don't really need to worry about it, and it lets you develop anywhere your laptop can go, and then is ready to be moved over in place when you are ready for production.
For custom themes and functional plugins, I use WP Pusher, and webhooks to notify the sites, so I develop in a dev branch or something, but all pushes to master are also synced to its respective site, no extra steps for me.
With this workflow, starting a new site takes me about 5 minutes, most of which is spent making a cup of tea while VVV is installing a brand new, isolated WordPress instance for me.