Risk when attacker has only physical access to a computer's USB port

Attacker has only access to USB port and can either attach a device or pen-drive to a computer, that can be on or off. Imagine a scenario where non-reliable people have access to offices, either because they clean after-hours (off scenario) or because they sit near the running computer (on scenario) but only that, what is the risk to this computer? Could you connect a pen-drive to it when the computer is turned off and run some malicious software on it? If the computer is running, can you connect a cable to this USB port and extract some information?