That's a big phone. That said, I felt the iPhone 5 was too big at first but I'm comfortable using it now.
Until they sort the input lag out though, it's nothing I'd ever consider. A knowledgeable mate of mine explained to me why Android devices appear to have this issue in laymans terms for me:
The reason Android is "laggy" is because all the UI processing (animations/transitions/etc) is rendered on the main system thread. The upside of this is that in theory there should be slightly more processing grunt available to the currently active app as anything the app needs takes priority over UI rendering.
The reason iOS and Windows Phone/Windows RT manage to stay fluid whilst doing tasks is that their UI rendering is processed in a separate worker thread. In iOS that separate thread is reserved for all Core Animation graphics rendering (which includes the UI). This means the currently active app can carry on with its work whilst also allowing the UI to feel smooth but the downside of that is the processing power is now divided. In reality, today's modern ARM processors can handle multiple threads quite comfortably so you might as well design your OS with separate threads for different tasks.
As of Android 4.1 (AKA Jelly Bean), Google has implemented UI rendering in a separate thread (internally they called this Project Butter) so Android should not suffer from UI lag. As OEM's like Samsung and HTC put their own "launcher" over the top of stock Android this can still cause lag as it's up to those OEM's to make their launchers multi-threaded. Nexus devices should be buttery smooth running Jelly Bean or later.
Makes sense now, and I would like to give a Nexus 4 a go.