~~~
I love how it annoys people who feel that you *must* fill in such pointless details in profiles. Does it really matter? No. Do people think it matters? Yes. There's the problem. If Twitter profiles asked what your favourite pizza was, I'm sure 90% of users would fill it in.
~~~

In the case of Twitter, avatars (be it a portrait photo or a completely different image) are very useful to visually identify who wrote a particular tweet in my timeline. You have a Gravatar — why not just use that as your Twitter avatar as well?
I follow about 200 people; if none of them had an avatar, chances are they would share the same default avatar, and reading my timeline would be a pain.

The hypothetical “favorite pizza” field is something completely different.

TL;DR Adding an avatar makes it easier for your followers to identify you and your tweets. I’d say that does matter.