The Victory could add the nutty taste but in that small of percentage I would think it would be very subtle. What Pale Malt did you use? Some pale malts can have a nutty a flavor, much more so than say a brewers malt. I used Rahr Pale Ale malt before and it definitely does have a nutty taste, much more so than Weyermann or Dingemans. Briess pale ale malt also has a nutty flavor.
This is one of the reasons I switched to Brewers Malt vs pale ale and pale ale has a more grainy taste than the lighter brewers malt. In brews that need pale ale malt I always go for Maris Otter.