That was the 2007-08 season which was a bit of a disaster in us attempting to get back into the League. It was a season where we never truly looked like going up or down - oddly, we've had very few of those in the last 15 years. 2012-13 runs it close for a season of nothingness. I'm sure that a season that had little riding on it from November onwards stymied any reason for more fairweather fans to go.
Our average attendance that season was 4,553 with a high of 5,900 against Crawley on Boxing Day. We even had two sub-4000 home attendances in early January against Altrincham and Weymouth.
Compare and contrast with the 2009-10 season that was competitive. The first round FA Cup game against Yeovil (who were two divisions higher) saw 6,144 attend. The average home attendance across the season was 5,730 (up 1,177 from 2007-08) with two 10,000+ matches (Luton and Rushden SF play-off) and just the one League attendance below 5,000 (a deadish rubber match with Wrexham at the end of the season on a Tuesday night).
Things are miles better now than in the Conference days.