I had some comms with crown cams, where my lifters (and springs) are from. They said that they haven't had issues like this before and the oil I'm using now and have been using during initial start up and break in, shouldn't be an issue. So might not be lifters.
Once I get my cam cover back from the painter shop I'll move the car inside and take a look at the top end while it's running.