The accident happened at about 0930 GMT at Pratten's Bowls Club in Midsomer Norton, near Bath.
Initial fears of a third fatality proved unfounded. No other aircraft and no-one on the ground were involved, investigators said.
An investigation was launched by the Air Accidents Investigation Branch (AAIB), with a team of experts making their way to the scene.
Det Insp Mike Williams, of Avon and Somerset Police, said: "An air crash investigation team are currently looking at the circumstances of this incident to establish the cause.
"We would like to speak to anyone who may have seen the balloon in the air before the incident, or witnessed the balloon come to the ground."
Witnesses said flames were seen coming from the basket.
Sarah Andrews said: "I could hear a flapping noise and looked up and there was a balloon. The canopy was collapsing in on itself; it was on fire on one side and the basket was on fire. It was coming down fairly quickly.
It was the first fatal balloon crash in the local area for 40 years "I thought it was about to come through the roof of the house, but it moved on a couple of hundreds yards and thankfully came down in a field away from houses."
She said it was terrifying to watch.
"As soon as it hit the ground, there was no way anyone was going to survive that. The fire was really intense and the gas canisters on the side started to go as well."
Robert Biggs, who was walking with friends, said: "It first sounded like a plane in trouble. It was more of a whistling sound. We looked up, there was a break in the cloud and this hot air balloon came hurtling down."
He said the basket was not intact when it crashed and the canopy was trailing up.
"It came straight down about 400 yards from us. It came down and there it was slap bang in the middle of the bowling green," Mr Biggs said.
Avon Fire Service said the balloon was being monitored by Bristol Air Traffic Control and was believed to have been attempting a high altitude flight at about 20,000ft.
There was a ground crew, containing some family members, who lost the balloon in cloud but arrived on scene soon after the crash.
Two off-duty firefighters from Bath were on a bike ride nearby and saw the balloon but the fire was well developed and they were unable to do anything.
The balloon also contained four liquefied petroleum gas (LPG) cylinders which were venting gas. Crews used three high pressure hose-reels to extinguish the fire.
Don Cameron, who owns Bristol-based Cameron Balloons, said it was difficult to speculate what could have gone wrong, but it was the first fatal balloon crash in the local area for 40 years.
Mr Cameron, who was the race director of last year's ill-fated Gordon Bennett Cup, in which two US competitors went missing and were later found dead off the coast of Italy, said the ballooning community would be in "total shock".
"Everyone who flies balloons in this area are known to each other. It's a dreadful business at the start of the New Year," he said.
Send your pictures and videos to yourpics@bbc.co.uk or text them to 61124 (UK) or +44 7725 100 100 (International). If you have a large file you can upload here.
Read the terms and conditions
