Using Conditional Formatting with time

Got a spreadsheet that I'm trying to use conditional formatting on the cell B3 to H3 inclusive.

I want the row to change colour if hours are entered in D3, so I've created the following in conditional formatting:

=$D3>TIME(1,0,0)

However, there are rows without any information on and I want those to remain the neutral white until hours appear in column D but the above formula seems to want to colour these empty rows too.

Any help will be greatly appreciated.

Thanks