It's not just Kenya. Squaring up to the seamier side of empire is long overdue

History teaches us that empire can bring out the worst in people. In Britain we applaud the "civilising mission" of our imperial past, but are less happy to acknowledge the violence and brutality that so often girded our imperial endeavour. It is time we were more honest.

As a nation Brits nurture memories of empire that are deceptively cosy, swathed in a warm, sepia-tinted glow of paternalistic benevolence. The British empire, so the story goes, brought progress to a primitive and savage world. Education, hospitals and improved health, steamships, railways, and the telegraph – these were the tools of empire, brought to colonised peoples by the gift of commerce and good British government.

Illustration by Daniel Pudles

We take pride in this imperial heritage, pointing with scorn at the lesser achievements of other European powers – the French, Italians, Germans, Belgians and Portuguese – whose empires we variously view as haplessly mismanaged, malignly exploitative and brutally coercive. Britain's empire was better than all the others, historians such as Niall Ferguson, Andrew Roberts and Lawrence James have assured us, so why should we worry?

The reasons to worry became all too apparent last week, in a path-breaking judgment of the high court: Mr Justice McCombe ruled that the British government has a case to answer in relation to charges of systematic torture and abuse of detainees during Kenya's Mau Mau rebellion of the 1950s.

We have long known that Kenya was a dirty war and that bad things happened. But the extent of abuse now being revealed is truly disturbing. Documents brought to light in connection with the Mau Mau court hearing catalogue more than 400 separate charges of abuse, spanning every element of the British security and administrative services in Kenya at the time.

And the matters raised are far from trivial. Of the four elderly Kenyan plaintiffs who brought this case, two were allegedly the victims of castration, one claims to have been savagely beaten and left for dead on a mortuary slab, and another was allegedly the victim of repeated sexual abuse – all acts conducted during British "interrogation" of suspects against whom no crime had been proved.

In Kenya there has long been indignation at British cant in refusing to acknowledge that such things happened. The sense that Britain has tried to deny Kenyans their own history by removing documents and concealing them in the bowels of the Foreign Office for more than 50 years has only deepened these resentments. All of this will be aired in the high court early next year, when the Kenya case will be heard in full. It is going to be very uncomfortable for those in the Foreign Office who have tried to prevent this case coming to court – and for many in the British political establishment who are still in denial about the realities of our imperial past.

And the problem goes far beyond Kenya. In a further revelation, it has been admitted in the House of Lords that the Foreign Office "irregularly" holds 9,500 files from 36 other former British colonies. Do these hold further horrors yet to be revealed of colonial misdeeds? The discovery of this vast tranche of documents has prompted historians to suggest that a major reappraisal of the end of Britain's empire will be required once these materials have been digested – a "hidden history" if ever there were one.

Squaring up to the seamier side of our empire is long overdue. However benevolent empires aim to be, they are invariably built on political, economic and military domination. Empires are by their very nature exploitative, the authority of imperial rule often established and sustained through violence and coercion. In all of this, Britain's empire was no different than any other.

To the imperialist at the time, such matters were functional and may even have seemed immaterial; to the colonised, they marked out the experiences that mattered – the history of oppression, of occupation and of subjugation that shaped their national consciousness and, often, their political identity.

It is time for a reappraisal of our imperial past – for a new kind of reckoning that takes account of the power relations that must inevitably determine the history of any empire. This should not be just a balance sheet of progress, but rather a candid review of the history, warts and all.

And this is a history that we in Britain should all be more familiar with than we are. We need to begin in our schools. Though the history of the British empire is now available to students at GCSE and advanced levels, it is seldom taught. As history has shrunk in our schools, the diet of courses offered has come to resemble fast food – familiar topics, hastily thrown together from familiar ingredients: comfort eating. Can it be right that our schoolchildren are far more likely to learn about the Americans in Vietnam, or the Russian revolution, than they are about the British empire?

At university level, too, we can do much better than we do. The history of British imperialism ought to be part of the teaching of mainstream British history, part of an explanation of how Britain has become the society that it is in the 21st century. British students need to know how we lost an empire, just as they need to understand the place of former colonies in the modern world. We have much yet to do to rise to this challenge.

We will achieve this more readily if we can move toward an honest public debate on difficult imperial questions, such as the torture and abuse of detainees in Kenya. When revelations about these atrocities were first published in 2005, the reaction of many in the British establishment was to dissemble and deny. The bitter truth about torture was disguised behind arguments about death rates and the relative levels of abuse, as if such things mattered. At least Mr Justice McCombe has brought the denial to an end.

But can we now move forward to the more honest debate we need? Probing the imperial past can be an uncomfortable experience, but some of Europe's other great imperialists of the modern world have done so with greater honesty and humility than have the British. The German government has formally apologised for the enslavement and genocide of the Herero peoples of Namibia in the early part of the 20th century, and even the French have managed to swallow some national pride over the Atlantic slave trade – though not yet over Algeria.

We live in an age where the notions of atonement and historical reconciliation have become increasingly important in relations between countries. Sometimes the reasons for an apology can be transparently political, as David Cameron learned to his cost in Pakistan. Such flagrant opportunism aside, the British have so far remained implacably opposed to an honest assessment of their empire. It is time to face up to it.

We do not need to apologise to everyone for everything. But we do need to be willing to admit our imperial wrongs, and in the process to admit that others may have rights that we have infringed. To do so would mark a post-imperial maturity, a coming of age. It is time to lay the ghosts of empire to rest.