The point that broke it for me was the transition to book 2, when it introduces the second cast of characters which at that point seem to have nothing to do with the first set. Even though book 2 is still gardens of the moon, it really did feel like taking a full stop and picking up a whole other book for a while. It comes together fairly quick, but that first time it was just too much of a wall to start a new book inside the new book I just started.
It took me two tries to muster the interest to struggle on, but was well worth it. Ironically enough, the PoV at the start of book 2 I had so much trouble with is one of the more entertaining characters in the series.