Mailing List Archive: Multi thread reading a file

Mailing List Archive: Python: Python
Multi thread reading a file

Index | Next | Previous | View Flat

magawake at gmail

Jun 30, 2009, 6:52 PM

Views: 596
Permalink

Multi thread reading a file

Hello All,

I am very new to python and I am in the process of loading a very
large compressed csv file into another format. I was wondering if I
can do this in a multi thread approach.

Here is the pseudo code I was thinking about:

Let T = Total number of lines in a file, Example 1000000 (1 million files)
Let B = Total number of lines in a buffer, for example 10000 lines

Create a thread to read until buffer
Create another thread to read buffer+buffer ( So we have 2 threads
now. But since the file is zipped I have to wait until the first
thread is completed. Unless someone knows of a clever technique.
Write the content of thread 1 into a numpy array
Write the content of thread 2 into a numpy array

But I don't think we are capable of multiprocessing tasks for this....

Any ideas? Has anyone ever tackled a problem like this before?
--
http://mail.python.org/mailman/listinfo/python-list

Subject	User	Time
Multi thread reading a file	magawake at gmail	Jun 30, 2009, 6:52 PM
Re: Multi thread reading a file	gagsl-py2 at yahoo	Jun 30, 2009, 9:54 PM
Re: Multi thread reading a file	magawake at gmail	Jul 1, 2009, 4:41 AM
Re: Multi thread reading a file	ldo at geek-central	Jun 30, 2009, 10:07 PM
Re: Multi thread reading a file	magawake at gmail	Jul 1, 2009, 6:28 PM
Re: Multi thread reading a file	stefan_ml at behnel	Jun 30, 2009, 10:30 PM
Re: Multi thread reading a file	Scott.Daniels at Acm	Jul 1, 2009, 8:49 AM
Re: Multi thread reading a file	gagsl-py2 at yahoo	Jul 2, 2009, 3:10 AM
Re: Multi thread reading a file	stefan_ml at behnel	Jul 1, 2009, 9:50 PM
Re: Multi thread reading a file	rylesny at gmail	Jul 2, 2009, 7:10 PM
Re: Multi thread reading a file	http://phr.cx at NOSPAM	Jul 2, 2009, 7:20 PM
Re: Multi thread reading a file	rylesny at gmail	Jul 2, 2009, 8:09 PM
Re: Multi thread reading a file	http://phr.cx at NOSPAM	Jul 2, 2009, 8:15 PM
Re: Multi thread reading a file	gagsl-py2 at yahoo	Jul 2, 2009, 8:47 PM
Re: Multi thread reading a file	http://phr.cx at NOSPAM	Jul 2, 2009, 8:55 PM
Re: Multi thread reading a file	rylesny at gmail	Jul 2, 2009, 9:13 PM
Re: Multi thread reading a file	ldo at geek-central	Jul 4, 2009, 5:12 PM
Re: Multi thread reading a file	steve at REMOVE-THIS-cybersource	Jul 4, 2009, 6:35 PM
Re: Multi thread reading a file	ldo at geek-central	Jul 5, 2009, 2:52 AM

Index | Next | Previous | View Flat

Interested in having your list archived? Contact lists@gossamer-threads.com