SVG images can contain CSS, CSS are a possible injection point for XSS
See https://stackoverflow.com/questions/3607894/cross-site-scripting-in-css-stylesheets
And https://code.google.com/archive/p/browsersec/wikis/Part1.wiki#Cascading_stylesheets
Which both point out
As a little-known feature, some CSS implementations permit JavaScript
code to be embedded in stylesheets. There are at least three ways to
achieve this goal: by using the expression(...) directive, which gives
the ability to evaluate arbitrary JavaScript statements and use their
value as a CSS parameter; by using the url('javascript:...') directive
on properties that support it; or by invoking browser-specific
features such as the -moz-binding mechanism of Firefox.
So, yes, you can use user controlled SVG's to execute script in the domain of the host page.
Using Content-Security-Policy you can restrict where you expect to find styles or script. Another thing you could try is serve these SVG's in a sandboxed cross-domain iframe.