Party hails ‘new era’ with victory over pro-independence forces but fails to secure enough votes to govern alone