Short answer: This is a matter of preference for the most part, and as long as you stay consistent with your spelling, you should be fine with whichever you choose.
Long answer:
The root of this issue is transcribing Japanese names and words.
If you have a look at this
wikipage, you'll notice that Sess' name for example is written with a special character, an o with a line over it.
That would be
the most correct way to spell it. However when writing it is impractical to use since the character is not readily available on the keyboard and would have to be separately added or copy pasted into the text.
The o in question in Sesshoumaru's name and in the word
youkai is
a long o, and it is pronounced differently from the regular o we see in Kagome's name or in the word
miko
Sometimes the vowel length could even affect the meaning of the word:
hoshi with a regular o means a star, whereas
houshi with a long o means a monk (and is sometimes used to address Miroku in fics).
That is why I personally prefer to use the "ou" spelling (except in names like Tokyo or Kyoto that have a regulated English spelling).
I like to differentiate between the regular and the long o.
Of course in the context of your story none of this hardly matters as you're writing it in English anyway, not Japanese, so if you prefer to spell things with just "o" for simplicity's sake, there's nothing wrong with that.
Also the reason the long o is transcribed as "ou" (though sometimes you can see also variations of "oh" or "oo") is because when writing in the Japanese kana, "ou" is used to denote the long o, so it derives straight from there.
And of course, "ou" is much easier and quicker to use than the special character, the o with a line on top.
So there, that is the gist of it, but like I said it's largely down to your personal preference of how to transcribe Japanese names and words into English.
Inconsistency is what usually throws the reader off the most though! So whichever spelling you do decide on, try and stick to it even though it can be difficult sometimes.
Here are some of the names/terms featuring the long o in Inuyasha:
youkai/yokai
hanyou/hanyo
youki/yoki
Sesshoumaru/Sesshomaru
Shippou/Shippo
Kouga/Koga
Souta/Sota
Kikyou/Kikyo
Inu no Taishou/Inu no Taisho
Myouga/Myoga
Toutousai/Totosai
