Twiddle one`s thumbs

To not be busy, to not be working.

We twiddled our thumbs all morning and did not get any work done.