Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Python
Multi thread reading a file
 

Index | Next | Previous | View Flat


magawake at gmail

Jun 30, 2009, 6:52 PM


Views: 596
Permalink
Multi thread reading a file

Hello All,

I am very new to python and I am in the process of loading a very
large compressed csv file into another format. I was wondering if I
can do this in a multi thread approach.

Here is the pseudo code I was thinking about:

Let T = Total number of lines in a file, Example 1000000 (1 million files)
Let B = Total number of lines in a buffer, for example 10000 lines


Create a thread to read until buffer
Create another thread to read buffer+buffer ( So we have 2 threads
now. But since the file is zipped I have to wait until the first
thread is completed. Unless someone knows of a clever technique.
Write the content of thread 1 into a numpy array
Write the content of thread 2 into a numpy array

But I don't think we are capable of multiprocessing tasks for this....


Any ideas? Has anyone ever tackled a problem like this before?
--
http://mail.python.org/mailman/listinfo/python-list

Subject User Time
Multi thread reading a file magawake at gmail Jun 30, 2009, 6:52 PM
    Re: Multi thread reading a file gagsl-py2 at yahoo Jun 30, 2009, 9:54 PM
        Re: Multi thread reading a file magawake at gmail Jul 1, 2009, 4:41 AM
    Re: Multi thread reading a file ldo at geek-central Jun 30, 2009, 10:07 PM
        Re: Multi thread reading a file magawake at gmail Jul 1, 2009, 6:28 PM
    Re: Multi thread reading a file stefan_ml at behnel Jun 30, 2009, 10:30 PM
    Re: Multi thread reading a file Scott.Daniels at Acm Jul 1, 2009, 8:49 AM
        Re: Multi thread reading a file gagsl-py2 at yahoo Jul 2, 2009, 3:10 AM
    Re: Multi thread reading a file stefan_ml at behnel Jul 1, 2009, 9:50 PM
    Re: Multi thread reading a file rylesny at gmail Jul 2, 2009, 7:10 PM
    Re: Multi thread reading a file http://phr.cx at NOSPAM Jul 2, 2009, 7:20 PM
        Re: Multi thread reading a file rylesny at gmail Jul 2, 2009, 8:09 PM
    Re: Multi thread reading a file http://phr.cx at NOSPAM Jul 2, 2009, 8:15 PM
        Re: Multi thread reading a file gagsl-py2 at yahoo Jul 2, 2009, 8:47 PM
    Re: Multi thread reading a file http://phr.cx at NOSPAM Jul 2, 2009, 8:55 PM
        Re: Multi thread reading a file rylesny at gmail Jul 2, 2009, 9:13 PM
    Re: Multi thread reading a file ldo at geek-central Jul 4, 2009, 5:12 PM
        Re: Multi thread reading a file steve at REMOVE-THIS-cybersource Jul 4, 2009, 6:35 PM
            Re: Multi thread reading a file ldo at geek-central Jul 5, 2009, 2:52 AM

  Index | Next | Previous | View Flat
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.