removing constructional hearth = brick infill collapse?

[andtryagain]

Hi all,

I'm currently renovating a ground floor suspended timber floor of a Victorian semi. I've just started removing a constructional hearth that was in front of a bricked-up fireplace, but after removing the concrete slab I realise that the earth underneath seems to also be supporting the brick infill of the chimney breast. I.e although the sides of the breast are supported by brick pillars down to the subfloor, the brick infill section is seemingly only supported on packed earth

Is it OK to remove this earth (to allow new joists, insulation, air circulation), or do I need to keep it / put in a supporting lintel / build a supporting wall to the ground? I can't tell if the infill was toothed in to the breast on either side without smashing the plaster off...

Thanks for any thoughts!