This summer has been a grim reminder of the terrible consequences of failed drug policy in our prisons and on our streets.

Young people dying from prolific and uncontrolled novel psychoactive substances, prison rioting over rushed smoking bans and an alarming escalation of deaths from fentanyl (an opioid pain medication) in contaminated heroin were all both entirely foreseeable and foreseen consequences of woefully ineffective government policy.

A teenager died at Reading Festival last month of suffocation, quite possibly attempting to use a plastic bag to inhale nitrous oxide. This previously quite niche drug, known in the tabloid press as hippy crack, was supposed to have been banned by the Psychoactive Substances Act but, as expected, the increased profits available due to the reduction in licit supply have created irresistible incentives to supply the drug – and as a result we have seen its popularity surge. Nitrous oxide is now more popular than ever.

On our streets we have seen a continuing increase in the use of spice, these dangerous chemicals have been pushed out of high street shops only to be left openly dealt by violent drug dealers in Manchester Piccadilly Gardens and besides the Michelin starred Hakkasan restaurant in Soho as well as town and city centres across the country.

Along with the rising harms associated with the use of spice, we’ve seen rapidly rising rates of heroin contaminated with fentanyl. Fentanyl, an opioid a hundred times more potent than heroin, has killed hundreds of thousands of users in North America. Its low cost, high potency and tiny size make it a perfect product for unscrupulous organised criminals. Most users want heroin but they get fentanyl.

But nowhere have our drug policy failures been more pronounced than in our prisons, where there has been an influx of fentanyl, one in three, prisoners admitting to spice-use in prison, up to 15% of prisons developing a drug problem whilst in prison, all-time highs of drug abuse and violence, spiralling debt for the many, and profits for the few.

Prisons are failing for the same reasons drug policy is failing; policy is not informed by the experience of prisoners and drug users. Too few are listening. Overall 8% of men in prison in England and Wales report developing a drug problem since they have been in prison. In prisons with the worst regimes this is as high as 14-16%. The prevailing dogma refuses to recognise the weight of personal experience, the reality of market forces, and the social roots of addiction.

Conservatives supposedly have a strong belief in self-reliance and personal responsibility, yet prisons, supposedly places to reform character, actively infantilise people and leave them devoid of purposeful activity.  Overcrowding, understaffing and an obsession with ineffective forms of security have left most of our prisons as brutal warehouses dominated by the drug markets which have consumed them.

The previously bold prison reform plans have faltered and are dangerously close to outright failure. At the heart is the drug issue. To understand why prisons are failing you need to understand how people stripped of purpose; activity and social connection are so profoundly vulnerable to addiction and how no matter what security measures are taken, a route of supply will always be found.

Drug markets have evolved to supply new drugs to circumvent enforcement and testing systems. They have evolved new methods of supply such as the dark net and embraced old methods such as corrupting officials to ensure a ready supply of dangerous substances to the vulnerable to addiction. Now more than ever we need to focus on reducing the demand for drugs as supply reduction methods become increasingly more expensive and less effective.

This evolution of prison drug markets is part of a wider issue with shifting drug markets necessitating new policies. Over the last 10 years organic chemists have greatly accelerated the rate at which psychoactive substances are discovered. Business people have exploited the high-demand for popular illicit psychoactive substances by bringing these novel drugs into new markets, to meet demand with a legal supply of drugs with similar effects. Drug markets have changed irrevocably and prison policy must catch up. Policy must adapt or it will fail.

It was only last year that David Cameron was promising reforms to help the poorest in the country and giving a major speech promising the greatest prison reform in a generation. The political tide turns fast and now a focus on the poorest in the country from the current government seems unimaginable and the prospects of real prison reform seem more remote by the day.

The reality has been deep and rapid cuts affecting the poorest in our society the worst, and leaving a large cohort of vulnerable people moving between insecure housing, homelessness and often prison. This group has been ravaged by the harms of a dynamic and evolving drug market inside prisons, in deprived communities, and on the streets.

The cuts missed the opportunity to redefine the role of the state and focus on funding only effective state interventions. Instead we have underfunded prisons without addressing the demands on the prison service.

There are real opportunities for a pragmatic government to embrace better prison drug policy to reduce the burden on the Prison Service, reduce levels of problematic drug use and deliver safer communities. But the focus must be on making prisons a place of last resort where those willing to better themselves have that opportunity and those who aren’t willing are kept safe from harming not just the wider community but prison staff and prisoners.

The housing crisis, along with cuts that were too deep and too rapid to the criminal justice system, and increasingly counterproductive drug policies, have left the poorest in our society in peril. We need a renewed focus on helping the most vulnerable in our society.

At the core of failed prison policy and failed drug policy is a belief that there is salvation in punishment. We over-use prison because we think it reduces crime. It doesn’t. We over-use it because we think that doing so will reduce the pain of victims. It won’t.

Criminal justice policy is the most irrational policy area in our country. If we are to move towards cheaper and more effective policies we need to escape this irrationality. We need to restore the faith in experts by entrenching the experience of those affected by policy into policy development.

Groups who work with prisoners closely and listen to their concerns are not only invaluable in helping provide the support networks that are the essential prerequisite to turn lives around. They are also our best early warning system. These people knew about the spice crisis in 2012, long before it was picked up by the media, drug testing systems, politicians or the civil service. Building better drug policy in prisons and in wider society means empowering those experts and groups who can provide support and feed information into policy development.

As David Cameron said, somewhat hollowly, but with good logic, when he was Prime Minister: “Our economy can’t be secure if we spend billions of pounds on picking up the pieces of social failure and our society can’t be strong and cohesive as long as there are millions of people who feel locked out of it. So economic reform and social reform are not two separate agendas they are intimately connected to one another”

We face a vast but achievable task of building communities which protect themselves against the harms of drug abuse and build resilience to assaults from organised criminals seeking to profit from misery.