The 'Aggregate Function' is used to rollup the values when "Auto Group & Summarize" is set to yes for the query, it applies some logic depending on the datatype. The 'Rollup Aggregate Function' is used for summary values like totals in the report
For example:
Fred       $5
Fred       $15
Barney   $20
Barney   $30
Put both columns on your report,
Aggregate function tells Cognos what to do with them with each other, ie.if you specify Total, this will show...
'Aggragate function'-Total
Fred     $20
Barney $50
'Aggragate fuction'-Average
Fred     $10
Barney $25
'Rollup Aggragate Function' tells Cognos what to do when you want a "Rollup" at the end of the Report or a Grouping (usually a Total or Subtotal, but could be any of the functions.)
So, if Aggregate is set to Total and Rollup is set to Total you get this...
Fred        $20
Barney    $50
Summary $70
If Aggregate is set to Total and Rollup is set to Average you get this...
Fred $20
Barney $50
Smmary $35