The men, who are all from London and aged in their 20s, were arrested on Monday shortly after 1630 BST.
The arrests were made after Civil Nuclear Constabulary officers conducted a stop check on a vehicle close to the Sellafield site, in Cumbria.
The BBC understands the men were taking photographs and are all believed to be Bangladeshi.
The arrests are not thought to have been intelligence-led.
The men were held in Carlisle overnight and then moved to Manchester.
Four houses in east London were raided by counter-terror detectives as part of the investigation.
Scotland Yard said a small container was removed at one of the addresses.
The container is being examined. Explosives officers attended as a precaution, although the container is now not thought to have had explosives or hazardous material in it.
The North West Counter Terrorism Unit is leading the investigation.
Heavy security
The arrests were made within hours of the news breaking that Osama Bin Laden had been killed in Pakistan, and members of the public have been warned to remain vigilant amid fears of reprisals from groups sympathetic to al-Qaeda.
In a statement, Greater Manchester Police (GMP) said: "At this stage we are not aware of any connection to recent events in Pakistan."
A GMP spokesman said the investigation is in its early stages and no further information will be released yet.
A police officer can arrest anyone they "reasonably suspect" of being a terrorist, under Section 41 of the Terrorism Act 2000.
The Sellafield site is responsible for decommissioning and reprocessing nuclear waste and fuel manufacturing on behalf of the Nuclear Decommissioning Authority.
The site, which has operated since the 1940s, is heavily protected by both private security and officers from the Civil Nuclear Constabulary.
Sellafield has a bunker that stores about 100 tonnes of plutonium, which experts say is enough to manufacture thousands of nuclear weapons.
www.sublimedvds.com