After looking at how other shop systems do it I came across a guide for Prestashop that helped me figure it out: https://www.prestashop.com/en/blog/european-union-vat
You create the required tax rates per country, so eg DE standard, DE reduced, AT Standard, AT reduced.
You give your product the required tax rate of your home country.
You go to tax rules and create a tax rule called standard, reduced, etc. You modify it so that it reflects the tax rates you entered above per country.