Great question!
To me THRILLER is a film in which suspense is the main focus and keeping you at the edge of your seat through non-visual stimuli. By that I mean that a thriller's success is largely based on what is NOT shown rather than on what IS shown. Hitchcock really mastered this. It can also include funny incidents and some goofy situations to break the tenseness and give a sense of false security.
The lady vanishes, Rebecca, Marni, Vertigo are all examples of such a technique. Hitch also had a tendency to mix in lots of psychological treatment of his subjects; the cleptomania in
Marni, the Oedipus complex in
Psycho, the inferiority complex in
Rebecca, the bordeline necrophilia in
Vertigo. That of course involves quite a bit of sex, which is
implied. In a horror movie on the other hand such hints wouldn't be made; if sex were involved it would be rape or a sexual crime or something horrible like that.
Brian de Palma made lots of interesting and intriguing thrillers, with a more moden edge and less "censored" scenes (after all he created those about 20 years after Hitch's classics) at least at his first half of his career:
Body Double, Dressed to kill. Highliy watchable all of them.
Charade I would also categorise in this genre. It does have suspense and grace.
HORROR films on the other hand rely on making the viewer feel terror.
Terror is not the same as fear. We feel fear for something that may happen to us, something tangible, something that poses a realistic risk. Imagine if we see a burglar in our homes. This is fear. We fear for our lives, our right to do as we choose with our body.
But if we see a burglar in
another person's home from the window across the street and we actually see them kill the neighbour, then we feel terror. It has to do with empathy. So, the more we realise that, the more effective the horror becomes.
Hitchcock in his film with a similar theme never actually shows the murder, he hints at it and we have our doubts; so this is suspense, not terror. De Palma in his similar film Body Double also makes us double-doubt the scene by the 3/4s of the film. We do not feel terror, we feel suspense on what will happen and what the hero will do to unravel the plot. That's the main focus.
On the other hand in
Fright night,
http://www.imdb.com/title/tt0089175/we do see the vampyres kill the neighbours. We are terrified! They might do the same to us at one point!
The thing about
the Exorcist that has made it into an enduring success and a masterpiece of cinematic horror is that it actually manages to convince you that there IS evil and it is out to get us, maybe not today, maybe not tomorrow, but it might and we have witnessed that it spares no one, not even an innocent child.
That's terror.
There are also sub-genres however, like Splatter and Gore. Those rely on disgust and humiliation more than terror/horror.