I used to live in London and always needed to get back to the station quite sharpish, and my best advice is that you can't rely on the 1 or 5 buses to get you back on time if you are in a rush but want to stay for the full game, especially now the crowds are on the up.
As they're local buses and not football specific, you're at the mercy of Oxford traffic and having no idea when one is going to turn up. Also, the time it takes to travel down the Cowley Road into the City makes it feel like you're travelling to Mordor.
I would quite often get to the bus stop, wait ages for a bus before one would turn up followed by another a few moments later. It was then infuriating to have to wait for that bus I was on to fill up, when it could have just left to regulate the service and allowed the second bus to take on more of the passengers. So even when you get on one, it can take an absolute age to leave.
In the end I switched to taxis as I was missing too many trains - 001 taxis have a really good app and I used to pre-order one from Littlemore for 5pm to avoid the stadium traffic. Would always get me there in good time and only cost £10/£11. Obviously that's a bit of a luxury, but it was the only way I could guarantee getting to the station on Time without leaving early.