It was on my third visit to the Shirley Baker: Women and Children; and Loitering Men exhibition at Manchester’s Art Gallery when it all finally clicked with me. I knew what I liked as well as why I’d been having a problem. I’d been distracted by the eyes…

The eyes were of the children who Shirley Baker photographed around Salford during the 1960s, as Manchester was in transition from industrial Victorian powerhouse into an early version of the modern city we now know. Baker documented the demolition of the red-bricked terraced world. Her work captured the urban squalor of urban housing in Northern England but also life as it was then led. Her favourite subject was the street children, sub-Dickensian urchins in their oversized and undersized clothes, playing games despite (and sometimes amid) the half collapsed buildings and septicemic rubble. Yet what I found so striking and, indeed, surprising, was the number of children looking back at us; their eyes more captivated by the camera’s gaze than perhaps even the camera’s gaze was captivated by them.

There lay my problem. It was all about the eyes.

The children aren’t alone in this act of gazing back at us. The faces of men and women also look at us with bemusement, if not outright amusement. This is social documentary but of a kind where the role of the photographer is all too evident. The difference is the difference between street portraiture and that vaguely ambiguous term “street photography”. This exhibition is primarily about street portraiture and the portraits do tell us something about that world. Yet, at first glance, it is questionable how much they really tell us. One has to view them with a certain detachment and doubt the motives of not just the photographer but the subjects smiling into her camera. If you don’t ask why they smile, this can descend all too quickly into lazy sentimentality and crass judgements about happier days and a more naive world.

I was reminded, in all of this, of Bruce Davidson’s pictures of Wales, taken back in 1965, which were part of the particularly excellent Strange and Familiar exhibition which has just left Manchester. I’d felt similar qualms about the fixed poses, the stereotyped scenes, and the complicity of real people turned into actors. It was only later that I’d heard Garry Winogrand express his own visceral reaction to Davidson’s work, telling an audience of Rice University students that the work “is about his misunderstanding of [Diane] Arbus’s work. He thinks her work is strong because he thinks she collaborated with who she photographed.” The result, said Winogrand, was “morally sickening” and “all about what the white middle class liberals wants to think what the blacks are like.”

Not that Shirley Baker’s pictures provoke that kind of response, but some of this feels not dissimilar to how I responded to Davidson’s vision of Wales. I wasn’t quite sure how much of it to believe.

The temptation with Baker’s work, as well as any that turns a camera on an underclass, is that it might too easily be judged by its content; as a version of that modern tendency of some photographers who gravitate to vagrants because of their inherent drama and character. It’s really a danger of viewing art by content and not by form, as Susan Sontag once argued. “Real art has the capacity to make us nervous,” she wrote in Against Interpretation. “By reducing the work of art to its content and then interpreting that, one tames the work of art.”

Reducing Baker’s photographs to the subjects of the title can and perhaps does tame it.  Yet on that third visit, I realised something else. It was on my third visit that I ignored the sequence and didn’t follow the pictures as they led around the gallery walls. I was, by then, almost immune to the eyes of the children. I, instead, found myself loitering around groups of less celebrated photographs, moving against the flow of the crowd and searching for points of similarity.

Because if you can ignore the sad look of the women and children, and look beyond the baleful stares of the loitering men, you begin to realise that they’re surrounded by a world Baker reveals with haunting insight. The highlights, for me, are those photographs that aren’t so neatly described and where Baker’s understanding of photographic form are more evident and deserve greater credit. The impish smiles and devilish amusement of the street kids can distract from the pictures of the half demolished buildings and scattered architectural detail. Landscapes of crushed homes are punctuated by buildings standing like the last teeth on the cusp of a gaping dark maw. It’s the dark humour of the graffiti and also the pictures where the eyes of the subject are not diverted towards the camera. Suddenly the actors are actors no longer and are real people frozen in their proper context.

Baker has an eye for nuances and darkness, such as the squat dark building that stands with twin entrances both described by roughly painted arrows pointing at them and the word “shit” is repeated next to both. It’s as far from the smiling children as you could get and, in its way, it’s even more rewarding.

Part of the way around the exhibition is a map on which visitors are encouraged to pin flags denoting where they or their family lived in Salford. It’s a reminder that much of this exhibition is parochial. It is meant to evoke a world for people who once lived it. Many of the children gazing from the photographs have gazed on those pictures, visited the exhibition, and, indeed, some have contributed to the content. Yet this says nothing about the photographer and, really, the art is found in photographs as photographs and not as photographs as social realism.

This might sound like a highfalutin quibble but I hope it’s not. A lot of the exhibition reminds you that the photographer is present in the scene. Perhaps the strongest connections that emerge are between Baker and the people she made her life’s work.

As for the artist: that’s there too if you deeply enough. Look beyond the eyes and all impish charm and you will find moments of profound transparency. There you find a very powerful narrative of urban living, from a time before high rise blocks and long before the Grenfell Tower disaster was a sad reality. What that story tells us is a lesson about then but also about now and the role of an underclass physically, literally, but also psychologically trapped by architecture.

Shirley Baker: Women and Children; and Loitering Men, at Manchester’s Art Gallery until Monday 28 August 2017. Free