COM/XPCOM. Go dig into the Mozilla codebase.
Also, another vote for CSS. Mostly because its so hard to debug and figure out what exactly is making your page look like shit. Add to that varying browser support. Its hell.
Objective C... It is the weirdest language I've ever seen.
SML.
I remember at CMU, the sophmore level programming language theory course (the name escapes me, it was 15-212) was wildly different than anything I had done before. All programming courses one took up to that point were the normal intro level courses that most passed out of and a data structures/algorithms course. Sitting down the first day and being introduced to SML along with language constructs with no analogue in the programming languages I knew was tough. I struggled for about a month, much like I imagine many do when learning about pointers: you may be able to do some of the work, but you don't have a proper conceptual model to be able to reason about it. All of a sudden, after re-reading the material from the beginning, everything just clicked. It was probably my favorite undergraduate CS course from then on.
I have a couple for various reasons:
The hardest that I gave up on was Groovy. The language itself seems really nice, it was just hard because of the lack of documentation that I was able to find so I just switched to Scala since I had to learn it for a class anyways.
The hardest that I still use is Haskell. While there documentation is phenomenal compared to other languages, I have found that it isn't something that you can just pick up and start using and be able to write idiomatic programs without studying theory. For me, when I get to monads I struggled a bit. Then I get to the monadic bind and other operators like $ and . and I find I have to keep going back to tutorials to see when and how to use them properly.
Javascript. I've learned the basics of SML and Ocaml, and a bit more than basics of Haskell, but I just can't bring myself to learn how to use JS effectively.
More generally, I find it hard to learn how to do GUIs and pretty web pages.
For me it was (and still is) ORM, specifically Doctrine2. In principle it seems pathetically easy, but once the queries start getting immensely complicated ( Many-to-Many over multiple tables, with conditional form data, etc) I still find myself having to fallback to SQL.
Other contenders have been:
1. Bison ( tokens, parsing, syntax, was all difficult the first time around).
2. OpenGL because remebering function names for the api is RIDICULOUS! ahh glDrawElementsInstancedBaseVertexBaseInstance now what are the args again?
CSS
It is deceitfully simple at a glance, but to make a useful design with it is hard. There are so many hacks to get it to do what you want and hacks to work around bugs.
Haskell. While I've sung its praises on HN before, I am just a novice. I've gotten deep enough to see how great it is, but not far enough to write anything nontrivial.
Honestly, CSS. It's so different from programming that I can't wrap my head around it.
Ask me to write a simple web page and I'll have ~10 tabs open to w3schools.
Javascript.. I have been working with the DOM for the past 3 years yet I feel like I still do not have a solid grasp of Javascript.
Another vote for CSS, but then again I've never actually tried to use COM
Something about CSS prevents me from really grasping it and using it with any real skill. I mainly live off of designer friends and things I can cobble together with the help of google.
Recently, I'd probably say SAML. There are so many intricacies in the spec that it's difficult to know how best to apply it to a problem, and the implementation I use is poorly documented.
probably not most difficult to learn but most hated: Perl
most difficult to learn for an OO-guy: Erlang (and most interesting as well)
Seriously Android
Unreal would be my first choice. A company recently sued Epic for how crap it is but they lost. Still, the fact that they did sue them in the first place tells you a lot but I'm not allowed to have an opinion.
If I could sue a language I'd sue bash. I know it'd be futile but there is a reason that so many replacements exist but maybe a better reason why none of them have supplanted bash.