You should be able to reset the instance by typing '/reset' (no apostrophes) into your chat, and then attempt again, from within the dungeon.
If the quest itself is an issue, use L to open your Quest log, and find the quest. Highlight it by clicking on it, and if there's an 'abandon' or 'reset' then you can click on it. If it's grayed out, you cannot. At that point, you'd have to contact support;