Urgent action must be taken to ensure that emergency departments remain safe and sustainable, doctors have warned.
The College of Emergency Medicine said that A&Es were facing "intolerable pressures" after a poll found that 62% of emergency doctors reported their job is not sustainable in its current form.
The survey of 1,077 emergency medicine consultants also found that 94% regularly worked more than their contracted hours to help maintain levels of service.
The college said that this has "potentially serious repercussions" for safe working by senior medics.
It also warned that the pressure was reducing the attractiveness of the specialty to young doctors and causing difficulty in retaining doctors - some of whom are leaving the UK to work elsewhere.
The report shows that in 2009 just two emergency consultants emigrated from the UK but in 2013 the figure had already risen to 21 by August 8.
The report said: "The results show a worrying trend.
"Increasing numbers of consultants who have been trained by the NHS are choosing to use their skills abroad."
The report was published after figures showed that the number of A&E units failing to meet the Government four-hour target has almost trebled in a year.
Some 39 departments failed to meet the target of seeing 95% of patients within four hours during the period July to September, compared with 14 units during the same period in 2012, according to NHS England data released last week.
Dr Taj Hassan, vice president of the college and one of the authors of the report, said: " The college is working with its members and fellows to help them do all they can in this challenging situation but we need prompt action by relevant stakeholders on the three key recommendations in this report.
"Senior medical decision makers in emergency medicine provide one of the most vital strands in maintaining safety for emergency care systems in the UK.
"A failure to address these issues will compromise this ability and also further worsen the present workforce crisis affecting emergency departments."
Dr Andrew Goddard, of the Royal College of Physicians, said: "This survey reflects what most consultant physicians observe in their hospital's A&E departments with an unmanageable workload and difficult working conditions that make emergency medicine unattractive to trainees.
"Finding ways to make consultant working in hospital medicine more attractive where such conditions exist is one of the key challenges facing the NHS at the moment and needs to be addressed as a matter of priority."
A Department of Health spokesman said: "We know we need to do more to support emergency departments to do their important work through winter and beyond. That is why we are investing £500 million in A&E over the next two years and we tasked Health Education England with developing plans to encourage more medical students to become A&E doctors in the future.
"We have asked Professor Sir Bruce Keogh to carry out a review to look at the demands on services and how the NHS should respond. Sir Bruce is due to report and share his recommendations very shortly.
"We are also investing in other parts of the health service, to help keep people healthier and out of A&E departments in the first place."
Dr Paul Flynn, chairman of the British Medical Association's consultants committee, added: "Consultants working in emergency medicine face some of the most challenging, high-pressured and stressful work environments in the NHS, often with limited resources and gruelling workloads.
"Unsurprisingly, the result has been fewer doctors choosing to go into emergency medicine and others leaving to work abroad, meaning existing consultants are working flat out to meet rising demand.
"We urgently need to look at how we can make working practices in emergency medicine safe and sustainable to address this recruitment and retention crisis."