How many amps do you really think you'll need in that barn? If its less than 40, then what you propose should be fine (and I'd consider a 40A breaker at the house to enforce it). But if you'll be pushing 50 amps or more, then you need to upsize the wire to stay within a 3% voltage drop.
If you know you have some 240V only loads, then the neutral can be downsized, but I wouldn't go down more than 1 size). Since you're already upsizing for voltage drop, I don't think you'll ever have an ampacity problem on the neutral just maybe excessive voltage drop depending on how well you balance things.
If you can get a discount for buying a whole spool of wire (and you can at Lowes and Home Depot), I'd suggest getting 500' of #2 copper and 500' of #4 copper. Use the #2 for the two ungrounded power feeds and the #4 for neutral and ground. Or you could go aluminum in #1/0 and #2 which may be substantially cheaper and easier to lift too. You could even buy URD wire and skip the conduit. But if you can afford it, using a conduit and THWN conductors is a nicer install as it allows for change/replacement.
The barn is far enough from the house that you may want to consider a 3 wire feeder instead of 4 wires (it will save the cost of 1 wire, and you'll need to bond the panel to neutral). But you can't if you have any metal pipes or other grounded metal wires (phone, TV) to that barn from the serving structure.