I think I might have figured it out! I think it might be the harmonic balancer pulley. That would explain why the sound remained after belt removal.

I'll check that again tonight after work, this is the video that makes me think so:

https://www.youtube.com/watch?v=HvWwCEyWSx0