Tjyylli

Here's an optimized way to parse the file with re, first taking the ID and then parsing the data tuples. This takes advantage of the fact that file objects are iterable. When you iterate over an open file, you get the individual lines as strings, from which you can extract the meaningful data elements.

import re

import pandas as pd



SEP_RE = re.compile(r":s+")

DATA_RE = re.compile(r"(?P<term>[a-z]+)s+(?P<weight>d+.d+)", re.I)





def parse(filepath: str):

    def _parse(filepath):

        with open(filepath) as f:

            for line in f:

                id, rest = SEP_RE.split(line, maxsplit=1)

                for match in DATA_RE.finditer(rest):

                    yield [int(id), match["term"], float(match["weight"])]

    return list(_parse(filepath))

Example:

>>> df = pd.DataFrame(parse("/Users/bradsolomon/Downloads/doc.txt"),

...                   columns=["Id", "Term", "weight"])

>>> 

>>> df

   Id     Term  weight

0   1    frack   0.733

1   1    shale   0.700

2  10    space   0.645

3  10  station   0.327

4  10     nasa   0.258

5   4   celebr   0.262

6   4    bahar   0.345



>>> df.dtypes

Id          int64

Term       object

weight    float64

dtype: object

Walkthrough

SEP_RE looks for an initial separator: a literal : followed by one or more spaces. It uses maxsplit=1 to stop once the first split is found. Granted, this assumes your data is strictly formatted: that the format of your entire dataset consistently follows the example format laid out in your question.

After that, DATA_RE.finditer() deals with each (term, weight) pair extraxted from rest. The string rest itself will look like frack 0.733, shale 0.700,. .finditer() gives you multiple match objects, where you can use ["key"] notation to access the element from a given named capture group, such as (?P<term>[a-z]+).

An easy way to visualize this is to use an example line from your file as a string:

>>> line = "1: frack 0.733, shale 0.700,n"

>>> SEP_RE.split(line, maxsplit=1)

['1', 'frack 0.733, shale 0.700,n']

Now you have the initial ID and rest of the components, which you can unpack into two identifiers.

>>> id, rest = SEP_RE.split(line, maxsplit=1)

>>> it = DATA_RE.finditer(rest)

>>> match = next(it)

>>> match

<re.Match object; span=(0, 11), match='frack 0.733'>

>>> match["term"]

'frack'

>>> match["weight"]

'0.733'

The better way to visualize it is with pdb. Give it a try if you dare ;)

Disclaimer

This is one of those questions that demands a particular type of solution that may not generalize well if you ease up restrictions on your data format.

For instance, it assumes that each each Term can only take upper or lowercase ASCII letters, nothing else. If you have other Unicode characters as identifiers, you would want to look into other re characters such as w.

You can use the DataFrame constructor if you massage your input to the appropriate format. Here is one way:

import pandas as pd

from itertools import chain



text="""1: frack 0.733, shale 0.700, 

10: space 0.645, station 0.327, nasa 0.258, 

4: celebr 0.262, bahar 0.345 """



df = pd.DataFrame(

    list(

        chain.from_iterable(

            map(lambda z: (y[0], *z.strip().split()), y[1].split(",")) for y in 

            map(lambda x: x.strip(" ,").split(":"), text.splitlines())

        )

    ), 

    columns=["Id", "Term", "weight"]

)



print(df)

#  Id     Term weight

#0  4    frack  0.733

#1  4    shale  0.700

#2  4    space  0.645

#3  4  station  0.327

#4  4     nasa  0.258

#5  4   celebr  0.262

#6  4    bahar  0.345

Explanation

I assume that you've read your file into the string text. The first thing you want to do is strip leading/trailing commas and whitespace before splitting on :

print(list(map(lambda x: x.strip(" ,").split(":"), text.splitlines())))

#[['1', ' frack 0.733, shale 0.700'], 

# ['10', ' space 0.645, station 0.327, nasa 0.258'], 

# ['4', ' celebr 0.262, bahar 0.345']]

The next step is to split on the comma to separate the values, and assign the Id to each set of values:

print(

    [

        list(map(lambda z: (y[0], *z.strip().split()), y[1].split(","))) for y in 

        map(lambda x: x.strip(" ,").split(":"), text.splitlines())

    ]

)

#[[('1', 'frack', '0.733'), ('1', 'shale', '0.700')],

# [('10', 'space', '0.645'),

#  ('10', 'station', '0.327'),

#  ('10', 'nasa', '0.258')],

# [('4', 'celebr', '0.262'), ('4', 'bahar', '0.345')]]

Finally, we use itertools.chain.from_iterable to flatten this output, which can then be passed straight to the DataFrame constructor.

Note: The * tuple unpacking is a python 3 feature.

Assuming your data (csv file) looks like given:

df = pd.read_csv('untitled.txt', sep=': ', header=None)

df.set_index(0, inplace=True)



# split the `,`

df = df[1].str.strip().str.split(',', expand=True)



#    0             1              2           3

#--  ------------  -------------  ----------  ---

# 1  frack 0.733   shale 0.700

#10  space 0.645   station 0.327  nasa 0.258

# 4  celebr 0.262  bahar 0.345



# stack and drop empty

df = df.stack()

df = df[~df.eq('')]



# split ' '

df = df.str.strip().str.split(' ', expand=True)



# edit to give final expected output:



# rename index and columns for reset_index

df.index.names = ['Id', 'to_drop']

df.columns = ['Term', 'weight']



# final df

final_df  = df.reset_index().drop('to_drop', axis=1)

Just to put my two cents in: you could write yourself a parser and feed the result into pandas:

import pandas as pd

from parsimonious.grammar import Grammar

from parsimonious.nodes import NodeVisitor



file = """1: frack 0.733, shale 0.700, 

10: space 0.645, station 0.327, nasa 0.258, 

4: celebr 0.262, bahar 0.345 

"""



grammar = Grammar(

    r"""

    expr    = line+



    line    = id colon pair*

    pair    = term ws weight sep? ws?



    id      = ~"d+"

    colon   = ws? ":" ws?

    sep     = ws? "," ws?



    term    = ~"[a-zA-Z]+"

    weight  = ~"d+(?:.d+)?"



    ws      = ~"s+"

    """

)



tree = grammar.parse(file)



class PandasVisitor(NodeVisitor):

    def generic_visit(self, node, visited_children):

        return visited_children or node



    def visit_pair(self, node, visited_children):

        term, _, weight, *_ = visited_children

        return (term.text, weight.text)



    def visit_line(self, node, visited_children):

        id, _, pairs = visited_children

        return [(id.text, *pair) for pair in pairs]



    def visit_expr(self, node, visited_children):

        return [item for lst in visited_children for item in lst]



pv = PandasVisitor()

result = pv.visit(tree)



df = pd.DataFrame(result, columns=["Id", "Term", "weight"])

print(df)

This yields

   Id     Term weight

0   1    frack  0.733

1   1    shale  0.700

2  10    space  0.645

3  10  station  0.327

4  10     nasa  0.258

5   4   celebr  0.262

6   4    bahar  0.345

Here is another take for your question. Creating a list which will contain lists for every id and term. And then produce the dataframe.

import pandas as pd

file=r"give_your_path".replace('\', '/')

my_list_of_lists=[]#creating an empty list which will contain lists of [Id Term  Weight]

with open(file,"r+") as f:

    for line in f.readlines():#looping every line

        my_id=[line.split(":")[0]]#storing the Id in order to use it in every term

        for term in [s.strip().split(" ") for s in line[line.find(":")+1:].split(",")[:-1]]:

            my_list_of_lists.append(my_id+term)

df=pd.DataFrame.from_records(my_list_of_lists)#turning the lists to dataframe

df.columns=["Id","Term","weight"]#giving columns their names

It is possible to just use entirely pandas:

df = pd.read_csv(StringIO(u"""1: frack 0.733, shale 0.700, 

10: space 0.645, station 0.327, nasa 0.258, 

4: celebr 0.262, bahar 0.345 """), sep=":", header=None)



#df:

    0                                          1

0   1                 frack 0.733, shale 0.700, 

1  10   space 0.645, station 0.327, nasa 0.258, 

2   4                 celebr 0.262, bahar 0.345 

Turn the column 1 into a list and then expand:

df[1] = df[1].str.split(",", expand=False)



dfs = []

for idx, rows in df.iterrows():

    print(rows)

    dfslice = pd.DataFrame({"Id": [rows[0]]*len(rows[1]), "terms": rows[1]})

    dfs.append(dfslice)

newdf = pd.concat(dfs, ignore_index=True)



# this creates newdf:

   Id           terms

0   1     frack 0.733

1   1     shale 0.700

2   1                

3  10     space 0.645

4  10   station 0.327

5  10      nasa 0.258

6  10                

7   4    celebr 0.262

8   4    bahar 0.345 

Now we need to str split the last line and drop empties:

newdf["terms"] = newdf["terms"].str.strip()

newdf = newdf.join(newdf["terms"].str.split(" ", expand=True))

newdf.columns = ["Id", "terms", "Term", "Weights"]

newdf = newdf.drop("terms", axis=1).dropna()

Resulting newdf:

   Id     Term Weights

0   1    frack   0.733

1   1    shale   0.700

3  10    space   0.645

4  10  station   0.327

5  10     nasa   0.258

7   4   celebr   0.262

8   4    bahar   0.345

Could I assume that there is just 1 space before 'TERM'?

df=pd.DataFrame(columns=['ID','Term','Weight'])

with open('C:/random/d1','r') as readObject:

    for line in readObject:

        line=line.rstrip('n')

        tempList1=line.split(':')

        tempList2=tempList1[1]

        tempList2=tempList2.rstrip(',')

        tempList2=tempList2.split(',')

        for item in tempList2:

            e=item.split(' ')

            tempRow=[tempList1[0], e[0],e[1]]

            df.loc[len(df)]=tempRow

print(df)