The Right Material Matters: The Role of Core Materials in Audio Transformer Performance