Pandas column of lists, create a row for each list element

I have a dataframe where some cells contain lists of multiple values. Rather than storing multiple values in a cell, I'd like to expand the dataframe so that each item in the list gets its own row (with the same values in all other columns). So if I have:

    import pandas as pd
    import numpy as np

    df = pd.DataFrame(
        {'trial_num': [1, 2, 3, 1, 2, 3],
         'subject': [1, 1, 1, 2, 2, 2],
         'samples': [list(np.random.randn(3).round(2)) for i in range(6)]
        }
    )

    df
    Out[10]: 
                     samples  subject  trial_num
    0    [0.57, -0.83, 1.44]        1          1
    1    [-0.01, 1.13, 0.36]        1          2
    2   [1.18, -1.46, -0.94]        1          3
    3  [-0.08, -4.22, -2.05]        2          1
    4     [0.72, 0.79, 0.53]        2          2
    5    [0.4, -0.32, -0.13]        2          3

How do I convert to long form, e.g.:

       subject  trial_num  sample  sample_num
    0        1          1    0.57           0
    1        1          1   -0.83           1
    2        1          1    1.44           2
    3        1          2   -0.01           0
    4        1          2    1.13           1
    5        1          2    0.36           2
    6        1          3    1.18           0
    # etc.

The index is not important, it's OK to set existing columns as the index and the final ordering isn't important.

    lst_col = 'samples'

    r = pd.DataFrame({
          col:np.repeat(df[col].values, df[lst_col].str.len())
          for col in df.columns.drop(lst_col)}
        ).assign(**{lst_col:np.concatenate(df[lst_col].values)})[df.columns]

Result:

    In [103]: r
    Out[103]:
        samples  subject  trial_num
    0      0.10        1          1
    1     -0.20        1          1
    2      0.05        1          1
    3      0.25        1          2
    4      1.32        1          2
    5     -0.17        1          2
    6      0.64        1          3
    7     -0.22        1          3
    8     -0.71        1          3
    9     -0.03        2          1
    10    -0.65        2          1
    11     0.76        2          1
    12     1.77        2          2
    13     0.89        2          2
    14     0.65        2          2
    15    -0.98        2          3
    16     0.65        2          3
    17    -0.30        2          3

PS here you may find a bit more generic solution

UPDATE: some explanations: IMO the easiest way to understand this code is to try to execute it step-by-step:

in the following line we are repeating values in one column N times where N - is the length of the corresponding list:

    In [10]: np.repeat(df['trial_num'].values, df[lst_col].str.len())
    Out[10]: array([1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3], dtype=int64)

this can be generalized for all columns, containing scalar values:

    In [11]: pd.DataFrame({
        ...:           col:np.repeat(df[col].values, df[lst_col].str.len())
        ...:           for col in df.columns.drop(lst_col)}
        ...:         )
    Out[11]:
        trial_num  subject
    0           1        1
    1           1        1
    2           1        1
    3           2        1
    4           2        1
    5           2        1
    6           3        1
    ..        ...      ...
    11          1        2
    12          2        2
    13          2        2
    14          2        2
    15          3        2
    16          3        2
    17          3        2

    [18 rows x 2 columns]

using np.concatenate() we can flatten all values in the list column (samples) and get a 1D vector:

    In [12]: np.concatenate(df[lst_col].values)
    Out[12]: array([-1.04, -0.58, -1.32,  0.82, -0.59, -0.34,  0.25,  2.09,  0.12,  0.83, -0.88,  0.68,  0.55, -0.56,  0.65, -0.04,  0.36, -0.31])

putting all this together:

    In [13]: pd.DataFrame({
        ...:           col:np.repeat(df[col].values, df[lst_col].str.len())
        ...:           for col in df.columns.drop(lst_col)}
        ...:         ).assign(**{lst_col:np.concatenate(df[lst_col].values)})
    Out[13]:
        trial_num  subject  samples
    0           1        1    -1.04
    1           1        1    -0.58
    2           1        1    -1.32
    3           2        1     0.82
    4           2        1    -0.59
    5           2        1    -0.34
    6           3        1     0.25
    ..        ...      ...      ...
    11          1        2     0.68
    12          2        2     0.55
    13          2        2    -0.56
    14          2        2     0.65
    15          3        2    -0.04
    16          3        2     0.36
    17          3        2    -0.31

    [18 rows x 3 columns]

using pd.DataFrame()[df.columns] will guarantee that we are selecting columns in the original order...

From: stackoverflow.com/q/27263805