How to average data by year?

I have a spreadsheet of about 8000 rows of data and I need to average a column of data by the year in the correponding column. I would like to be able to drag the formula down to cover all years in the dataset. Any help would be greatly appreciated.

Ex.

C D E
2007 2 Column of averages by year
2007 1
2007 3
2006 2
2006 4
2006 1
2005 2
2005 3

Thnaks DILIPandey,

This is great. How would you return only one annual average in the new column to graph? I entered the criteria manually, instead of referencing the year cell, for the first few years (1980,1981,1982) and selected those cells to pull down to continue the vector to the final year (2010). However, this just caused the entered years to repeat (1980,1981,1982,1980,1981,1982).

=AVERAGEIF($D$2:$D$8093,1980,$A$2:$A$8093)
=AVERAGEIF($D$2:$D$8093,1981,$A$2:$A$8093)
=AVERAGEIF($D$2:$D$8093,1982,$A$2:$A$8093)

Thanks DILIPandey, worked great.