I tend to prefer using the term "special physics" because that also covers strangenesses that aren't really a "system". And I feel like I heard that term before I started hearing "magic system", but perhaps that's because I came at this from a literature/writing perspective, rather than a gaming perspective? (Both perspectives converging on "speculative fiction".)