I agree, many of the big places to visit in the city are not quite right for young toddlers. Take a look for indoor playgrounds in your area. Some do birthday parties, but a lot also have open hours on weekends and for a fee your young one can go down slides, put on costumes, drive around in toy cars. I found some listed here http://www.toronto.com/searchResults?q=playground&r=category%3A%22Theme%20Parks%20%20Entertainment%20Centres%22. Some community centres also run open gyms (kindergym, they often call it) on the weekend, and kids can really be active in these venues. Another place to visit is Solar Stage http://www.solarstage.on.ca/, which runs plays for kids as young as 3. I'm personally a big fan of recreational and family swims at community centres as well, those can knock off an afternoon (but times can often conflict with naps).
Diane