Since this photo was taken (2010), the city grew exponentially around this fort. Toronto condominium boom dwarfed everything in sight.
Fort York was the site of battle of Upper Canada, where British troops were defeated by the American invaders. 700 men against 1700 men supported by 16 ships was not fair play, but war is war after all.
Americans then looted houses and businesses in the City of York as Toronto was then called, destroying parliament and government building and printing presses.