This from Maggie's tumblr hugely resonates with me on specifics.
An interesting journey artists must presumably all go on is towards the realisation that you are creating for certain people. Those people will love the book. Everyone else will not.
It takes time, but eventually you'll be fine with it.
* what a magnificently crazy word