On shared or VPS hosting I would not worry about using too much memory.
See, your shop creates cache files, those cache files, because they are php files and your server runs Opcache, are compiled to memory. They are stored in memory because memory has quicker read / write speeds than a drive does. So I would argue if your site is not maxing out the memory there is an issue.
On a dedicate server changes change a bit and a few more variables are added, but in your case you essentially want to max your memory out, it gives you the fastest site possible.