It is very easy to set up a VPS considering the tools now we have on our exposure. This post is more of a self note to myself, so pardon me for not explaining things.
To quickly setup latest Nginx, MySQL, PHP (with APC and Suhosin) & Exim, just use this script https://github.com/vladgh/VladGh.com-LEMP
These instructions will get things up and running a secure server.
Buy a VPS which will get you to root access to your server.
Create a new user with sudo powers which will be actually used to do everything on a regular basis since using root is not recommended.
adduser ashfame usermod -a -G sudo ashfame
Setup iptables (firewall)
From your new account, set up your key-based login, disable root login & password login so that only key-based logins are allowed.
sudo nano /etc/ssh/sshd_config PasswordAuthentication no PermitRootLogin no UsePAM no
Links which explain the above things in detail: