I dont know. It's just a bit strange, that it uses quite some time on category page. Maybe there is something wrong coded in the module. But Only the module dev can tell.
In general I believe that your server is ok and your configuration probably too. You could of course cache css and js files, that would probably help to.