The more time I spend working on DPS (first my hunter and now my shadow priest spec), the more I realize that DPS is built around a constraint model, far more so than healing is. Constraints, of course, are limitations on the system you're working with, either organically arising from the nature of the system or (as is always the case in games) artificially arising from the rules applied by the designers of the system.
Healing has some pretty simple constraints: cooldowns (global cooldowns and the cooldowns on abilities) and mana (and holy power, in the case of holy paladins). There are helpful mechanics that you can make use of, but those are more tools to help you survive within the above mentioned constraints. Without those constraints, healing would be a simple matter of casting your "best" (combination of power and speed) spell over and over (and that has happened at times in the history of WoW when designers didn't make the constraints compelling enough).
DPS would seem to be somewhat similar, but while cooldowns still exist as constraints (especially global cooldowns), the main power source (like mana) often doesn't. Healing decision-making revolves around selecting the right targets and dealing with the above mentioned constraints of cooldowns and mana. DPS decision-making revolves around a more precise prioritizing of abilities based on the constraints each ability places on every other.
That was probably as clear a winter's night in Seattle (Seattle is perpetually overcast and misting/raining during all seasons except a couple of months in summer, but especially winter) so let me explain what I mean by using the shadow spec as an example.
What Are The "Core" Abilities Anyway?
Here are the abilities that will make up the vast majority of shadow DPS:
- Mind Blast
- Devouring Plague
- Mind Flay
- Shadow Word: Pain
- Vampiric Touch
- Shadow Word: Death
- Mind Spike
For clarity, I'm only talking about single-target damage on boss-like creatures, not low-health adds. So Mind Spike will only be relevant if you take the talent From Darkness, Comes Light (FDCL). I've chosen that talent, so Mind Spike will be a part of the analysis. A similar exercise can be done with the abilities you gain from the other two talents on the tier (and the level 90 talents will add further constraints, though not particularly complicated ones it appears).
Mousing over those abilities, you can see some basic constraints, like mana costs and cooldowns. Those aren't the particularly interesting ones (mana is virtually impossible to run out of, in fact I can't drop below 95% mana or so--my healer heart weeps at the injustice), so let's get to the meat of the matter.
Constraints? You Mean, Just Cast The Best Ability Off Cooldown, Right?
That's the usual take on DPS and it'll work to some extent. This is the type of thing you'll usually see about the current shadow rotation, for example:
Shadow Word: Death > Mind Blast > Devouring Plague (with three shadow orbs) > From Darkness, Comes Light Mind Spike > Shadowfiend > Vampiric Touch = Shadow Word: Pain > Mind Flay
Basically, this means, use whichever of Mind Blast, Shadow Word: Death, three shadow orb Devouring Plague, instant Mind Spike or Shadowfiend is available, with that order of priority if multiple options are available. If none of them are and either Vampiric Touch or Shadow Word: Pain are near expiration, refresh them. If neither of the previous two sentences applies, use Mind Flay.
Doing this in a machine-like fashion won't yield terrible results, you'll do reasonable DPS...but it won't be optimal. To optimize, you need to juggle in your mind all the constraints I'm about to discuss in order to make the best choice on a moment-by-moment basis.
Let's get to it! (I'm going to omit mana costs and cooldowns, because we've already noted them. Also, remember that I'm discussing single-target damage on boss types of monsters, so I'll also ignore the duration of the damage-over-time [DoT] spells as a constraint.)
Benefits: The second hardest hitting shadow spell, provides a shadow orb
Constraint(s): None, right? Actually, the constraint is that you can only have three shadow orbs at a time
Shadow Word: Death
Benefits: The hardest hitting shadow spell, provides a shadow orb
Constraint(s): See above. Also, technically a constraint is that it can only be used when your target is below 20% health (unless glyphed) but we'll ignore that by lumping it into "cooldowns" since this constraint is irrelevant to what I'm describing
Benefits: A hard-hitting DoT spell, fantastic damage per casting time invested (DPCT)
Constraint(s): Requires three shadow orbs (technically, you can use any number of orbs, but you only want to use it with three)
FDCL Mind Spike
Benefits: Hits about as hard as Mind Blast (though it doesn't provide a shadow orb)
Constraint(s): You can only have two charges saved up, further procs will be wasted
Benefits: Better DPCT than the spells listed below
Constraints: None beyond the long cooldown
Benefits: Better DPCT than Mind Flay
Constraint(s): Cast time (this has greater relevance for a DoT than a conventional spell, I'll explain later)
Shadow Word: Pain
Benefits: Better DPCT than Mind Flay
Constraint(s): None, practically
Benefits: Better than a hole in the head (more correctly: much better than standing idle)
Constraint(s): None, practically
(I ignored mana returns as a benefit for a couple of abilities because, as I said, mana is close to irrelevant. You can price that benefit in for fights which contain as a gimmick taking away all your mana, ala Yor'sahj the Unsleeping.)
So these notes lead us to a logic that's a bit more complicated than the basic priority system laid out above.
Putting It Together In Practice
First some logical consequences...
There's little question you want to cast Mind Blast the moment it is off cooldown...unless you have three shadow orbs.
Which means that a Devouring Plague with three shadow orbs is the top priority...except that the only time pressure on Devouring Plague is the next Mind Blast. As long as you cast it before your next Mind Blast is available, there's no DPS loss.
Which can be handy if Shadow Word: Pain or Vampiric Touch is about to expire or you have two charges of FDCL Mind Spike.
That leads me to FDCL Mind Spike. It sits higher in the priority list than the DoTs...but since you can save two charges at a time, one charge of FDCL Mind Spike isn't a pressing priority. It's better to refresh a DoT that's near expiry before burning a charge of FDCL Mind Spike if you only have one charge.
But if you have two charges of FDCL Mind Spike, then you want to burn a charge of that off pronto, probably even above casting that three orb Devouring Plague.
Shadow Word: Death pretty much follows the same rules as Mind Blast when the target is low enough in health that Shadow Word: Death is available.
Shadowfiend should be used any time it's off cooldown and, since it has no constraints, could be used ahead of Devouring Plague or FDCL Mind Spike with one charge, but could also be pushed behind refreshing Vampiric Touch or Shadow Word: Pain.
This leads to some complicated decisions at times. Let's take an example:
Shadow Word: Pain is 2.5 second from expiring
Vampiric Touch is 2 seconds from expiring
You just completed a cast of Mind Blast, giving you a third shadow orb
A charge of FDCL Mind Spike just proc'd
The usual priority chart would have you cast Devouring Plague, then Mind Spike, then refresh the two DoTs. However, if you hold the Devouring Plague and Mind Spike in abeyance and first refresh Vampiric Touch and then the Shadow Word: Pain, then cast the Devouring Plague and then the Mind Spike, you'll see better DPS, since you have no opportunity cost for withholding the Plague and Spike for a few seconds, whereas the opportunity cost of holding back on the Shadow Word: Pain and Vampiric Touch is letting them expire (costing you DPS and procs from talents like From Darkness, Comes Light and Divine Insight which is further DPS).
This, however, creates a few more caveats...
The reason I mentioned using Devouring Plague before Mind Spike in the above example is because I did take the Divine Insight talent, which means that Mind Blast is always at risk of having its cooldown reset. Which means I actually do want to get Devouring Plague cast off as quickly as possible, so that if I suddenly have an unexpected Mind Blast to cast, I don't have to push it back to avoid wasting the shadow orb it would generate (and pushing Mind Blast back is a DPS loss). Which is a further constraint on Devouring Plague, if you use that talent. However, if you don't use that talent, then you'd actually want to cast the instant Mind Spike first...because your next Mind Blast will be coming at a predictable time (and you should be able to plan to get Devouring Plague cast before Mind Blast comes off cooldown) whereas you might get two quick procs of FDCL while you're casting Devouring Plague (and then waiting out the global cooldown it initiates) which would cause one to be lost if you already have a charge of FDCL (you can't hold three charges at a time).
Getting back to my earlier mention that cast time is a constraint when it comes to a DoT, if you had Vampiric Touch and Shadow Word: Pain close to dropping off (say ~2 seconds away) and both about equally close to dropping off, then you'd want to cast Vampiric Touch first. A cast time spell followed by an instant will lead to both spells being cast more quickly than the instant followed by the cast time spell, because the latter puts a GCD between the casts. Of course, in the case of the former, the GCD will still happen after both spells, so the total time used is identical...but since the lower priority of the DoTs means you'll generally be following it up with something non-mission critical (Mind Flay, oftentimes), you're better off pushing that follow-up spell back rather than adding extra time to the DoT refreshing if there's a risk that one of the DoTs could drop off your target altogether.
This is a pretty long wall of text and may or may not seem like a lot to bear in mind, but it all starts to feel pretty natural and instinctive once you've practiced on a dummy enough and put those constraints into practice.
The real point is that while basic priority systems give you a general idea of what matters more and what matters less, you need to look at the constraints of all your abilities, as illustrated above, to understand how to fit your abilities together in real time.