Rogues are a melee dps class. In pve it’s all kind of the same, ranged hit the boss with missiles from far away and the melee punch his ankles. In pvp classes play hugely differently.

How do I avoid taking over the tank’s role?
You just hit the mobs. Just because you’re in melee range doesn’t mean you’re a tank. You should be standing behind mobs (not absolutely necessary but you are more likely to hit), hitting them just like a mage would only it’s backstabs instead of frostbolts.

I can not use my stealthed attacks as I get a message “target is already in combat” or some such thing.
You are trying to sap the mob. Sap is basically like a melee polymorph that can only be used on mobs that are not in combat (you can be in combat, though, but then you’d have to be a subtlety rogue above level 80 to shadow dance). In dungeons the most effective way to dps is to never use any stealth abilities (does not apply for subtlety). Just open with mutilate from stealth (which will be free because of shadow focus). Keep slice and dice up and otherwise just spam mutilate and envenom (you have eviscerate instead right now).

You will get cheap shot and kidney shot, as well as garrote, blind, smoke bomb and stuff like that. If you need to (which you usually don’t in dungeons and can’t in raids because mobs are immune) you can effectively control a large number of mobs. Root one, stun two, blind one, gouge one, disarm one etc. Even though rogues play very differently from mages, they are built on much of the same playstyle. Lots of burst, avoiding being hit rather than tanking it, lots of control, very high requirements on reaction times and play to be good (a high skill floor is what this is called, and this part does not apply in most pve, though subtlety is a pretty challenging raid spec). Both classes also have hugely big potential and you won’t find yourself being limited by the class, rather you limiting the class a lot (again, mostly while soloing (challenging stuff when soloing then, not just derping 1 mob at a time) and pvp).

In pvp, your job is to control and pick off good targets with huge burst. You’re not a warrior, not supposed to be running right into the fight and trying to kill everyone. At low levels, you don’t really have much, though, so you will mostly be relying on burst to kill people.

The 3 specs play quite differently. Mutilate is the easiest spec in both pve and pvp. It has very good sustained damage in pvp (something the other specs lack, they rely purely on big burst) and is very straight forward. At 90, you will not be playing very differently to now. you use dispatch when you can (it’s an execute) and otherwise mutilate to gain combo points. You then keep up rupture with 5 points every time it falls off and apply slice and dice at the start of the fight with just a few points (envenom refreshes slice so you only have to do it once unless something goes horribly wrong – check out cut to the chase). When slice and dice and rupture are up, you’re just using envenom, trying to keep the buff from it up as much as possible.

Combat is an extremely spiky spec in terms of damage output. Much like assassination, you are keeping slice up (though it isn’t automatically refreshed) and rupture (though it’s a 1% dps increase and only on single target so it’s not necessary unless you’re trying to minmax). you are also keeping up a debuff on the target called revealing strike, which is just like your regular sinister strike only that it applies an important debuff. it also has an ability called blade flurry which makes your attacks hit nearby targets, really useful for moments with many enemies. Outside of cooldowns, the damage is low and the energy regen is low. You’re mostly standing around. Then you pop killing spree and your damage goes through the roof for a short moment. Afterwards you use shadow blades with adrenaline rush and your damage spikes again. You now have effectively infinite energy (regens so fast you can’t use it all), generate twice as many combo points and do a lot more damage. Then they wear off and we’re back in slow town. combat is kind of a weird one in pvp because of the length of the cooldowns and how much you rely on them. It’s fun and you can absolutely wreck someone when they’re up but then they are down and you’re pretty much running around for a while. One cool thing though is that you can do an eight second stun with kidney shot (compared to 6 for other specs), which is extremely long.

Subtlety is by far the most challenging spec in both pvp and pve. it’s my favourite because it is difficult, fast paced and bursty. in pve, you absolutely need to be standing behind the target to do damage (because ambush and backstab require you to be there) and you are juggling a bunch of buffs and debuffs just like before. this time it’s gone way overboard, you are monitoring rupture, hemorrhage, find weakness and slice and dice while keeping very close check on the cooldowns of shadow dance, vanish and preparation. the main idea of this spec is to keep “find weakness” up as much as possible. this makes you ignore all the target’s armour (50% in pvp) so you can see why it’s useful. this is the only spec that really relies on stealth (because your ambush applies find weakness) and being behind the target. in pvp it is very much a burst oriented spec, just like combat. the damage outside of find weakness is not very good. you can burst by getting a restealth (running away from combat and entering stealth again when combat drops) or by shadow dancing, which enables you to use all your stealth abilities in combat for 8 seconds. this is where the spec gets interesting, you can do a lot with this ability, stunning and silencing a whole group of people, sapping a druid that’s about to come help the monk you’re attacking or just spamming ambush on them until they fall over.

at your level, the specs are more or less the same and it doesn’t matter much what you’re playing, so i’d pick the one of these 3 you think seems the most fun. there are weapon requirements for the specs. for assassination you absolutely always need to be using two daggers (note that mutilate requires two daggers). for combat you need to be using a slow weapon (a mace, sword, axe or fist weapon) with a slow speed like 2.6 in your main hand (the one you have a war knife of power in now). for subtlety you need to be using a dagger in your main hand. combat and subtlety are both okay with having whatever in the offhand (the one with a heartache dagger). they will continue to evolve as you level up and become more and more like what i described above. most rogues will suggest leveling as assassination or combat, but i often vouch for subtlety because learning to play the difficult spec means that learning the other two will be really easy. subtlety is also a great leveling spec if you get nightstalker – you can just run up to the mob and ambush them. this kills them with heirlooms (not sure how much damage you do without) and if it doesn’t, you can follow up with an eviscerate. if the damage you do without heirlooms is not good enough to kill them within these 2 attacks, i’d play assassination instead.