I'm not so sure what it is about containers that is so appealing to babies. I recently filled a plastic containers with fridge magnets and lids for my one year old. She can play with it for quite some time simply pulling everything out and then putting it back in (or at least trying to!)
I loved The Activity Mom's idea here, and did a similar thing by putting fabric pieces in a wipes tub. (Does anyone else's children love to pull out all the wipes from the tub?!) This way the wipes are safe and my daughter still gets to pull something out of an extra tub.