Invisible Man is not your usual run-of-the mill book about race and identity. The unnamed narrator is different (a voice of the outsider - someone caught between many competing realities/ideologies), the language/style is VERY different (poetic in a kind of Jazz-Faulknerian hallucinatory kind of way), and the philosophical messaging is VERY different (Camus' the Absurd with a sprinkle of the anarchic?)
I can see why this book might be hard for a lot of people to read (for the reasons above) and perhaps also, because, for me, at least, beneath the dark humor and comedic irony, lurks a maelstrom of pain, and what it means to be Invisible. I believe Ellison quite intentionally left out "The" in his title of the book.