How to check if any value is NaN in a Pandas DataFrame
In Python Pandas, what's the best way to check whether a DataFrame has one (or more) NaN values?
I know about the function
pd.isnan, but this returns a DataFrame of booleans for each element. This post right here doesn't exactly answer my question either.
jwilner's response is spot on. I was exploring to see if there's a faster option, since in my experience, summing flat arrays is (strangely) faster than counting. This code seems faster:
In : df = pd.DataFrame(np.random.randn(1000,1000)) In : df[df > 0.9] = pd.np.nan In : %timeit df.isnull().any().any() 100 loops, best of 3: 14.7 ms per loop In : %timeit df.isnull().values.sum() 100 loops, best of 3: 2.15 ms per loop In : %timeit df.isnull().sum().sum() 100 loops, best of 3: 18 ms per loop In : %timeit df.isnull().values.any() 1000 loops, best of 3: 948 µs per loop
df.isnull().sum().sum() is a bit slower, but of course, has additional information -- the number of