They say that 133 people were rescued, but 16 are still missing after the four-deck Almirante went down near the popular resort town of Guatapé.
There were about 170 people on board. Witness Juan Quiroz told the BBC the boat sank in less than five minutes.
Leisure boats quickly came to the rescue, hauling victims from the water and the top deck of the vessel.
Some survivors complained that they had not been given life jackets.
Army helicopters and divers later joined the search-and-rescue operation.
Colombian President Juan Manuel Santos, who travelled to the scene on Sunday evening, said six people had been killed, revising down an initial death toll of nine given by officials.
More than 20 people have been taken to a hospital in Guatapé.
Eyewitness Louisa Murphy told the BBC: “We saw things flying off the side of the boat. And within, I think, probably about 20 second the boat had sort of sunk with just the top deck visible.”
Luis Bernardo Morales, a fire service captain involved in the rescue, said the boat was very close to the port when it sank.
“We do not know whether it was a mechanical failure, an overloading or something to do with the currents that caused it to sink,” AFP news agency quoted him as saying.
Local residents said that the vessel had sunk twice before but had then been repaired before being used again.
Guatapé is a popular Andean resort town, known for its water sports and recreational activities.