On May 9, 2004, 126 Colombians were captured during an attack on a farm near Caracas. Chavez immediately accused them of being foreign-backed paramilitary forces intent on violence. Overthrow his regime. These events have only further intensified the antagonism between pro- and anti-Chávez camps in Venezuelan society. Chavez also claimed that there was a coup plot to overthrow him in 2004. The authenticity of this coup has not been determined to this day. In October 2005, 27 of the accused Colombians were found guilty and the remainder were released and deported.

In early and mid-2003, a suffrage rights group called "Súmate" (Please Join) began collecting the millions of signatures needed to launch a referendum to recall the president. By August 2003 Some 3.2 million signatures were collected in March, but the electoral commission, dominated by Chavez supporters, refused to accept these signatures because the constitution stipulates that a recall can only be initiated after half of the presidential term (that is, three years). The collection began before Chávez completed three years in office. Opposition and international news reported that Chavez began punishing those who signed the petition, while Chavez's supporters claimed that many workers had been forced to provide signatures by their employers in the workplace. In November 2003, the opposition collected signatures again, collecting a whopping 3.6 million signatures in just four days. Riots broke out across the country amid rumors that Chávez had used deception to fool the signatory collectors.

Articles of the constitution require that signatures from 20% of voters must be collected before a recall referendum can be launched. In addition, the national identity card numbers of the signatories are not kept confidential. The opposition also accused the Chavez government of raising the threshold for signing by granting citizenship to illegal immigrants and refugees. The number of registered voters increased by 2 million before the referendum, significantly raising the threshold required to initiate a recall referendum by 20%. .

Numerous reports also allege that the Chavez government, including state-owned Petroleos de Venezuela, Caracas Metro, and companies controlled by Chavez loyalists, imposed penalties on those who publicly mailed signatures. ***Hospitals have been accused of arbitrarily firing employees who participated in the signing. Finally, after the opposition leaders submitted 2,436,830 valid signatures to the National Electoral Commission, the Electoral Commission announced a recall referendum on June 8, 2004. Chávez and his political coalition also began mobilizing voters to vote "no".

A recall vote was held on August 15, 2004, and 59% of the votes cast a "no" option. European Union observers claimed that the Chávez government imposed too many restrictions on them, preventing them from observing the election. The Carter Center observed that the election was fair and open. Critics claimed the results were false and pointed to documents proving otherwise, and many questioned whether the government-owned voting machines were genuine. Opponents claimed the election was a "massive fraud" and questioned the Carter Center's reasoning, but five polls conducted by the opposition also pointed to a victory for Chavez.