A public policy office usually means a group of people who influence laws or what the government does. For example, there may be a team of people who helps the state of Texas decide how best to use education money.
A government agency could be a lot of different things. They may be at the federal level (giant examples would be the FBI or the CIA), or there are also federal agencies like the Department of Agriculture or NASA. There may also be state agencies like the Colorado Department of Transportation that deals with the roads, buses, and trains. There are also very small agencies that may only have 20 people.