Mailing List Archive: finding most common elements between thousands of multiple arrays.

Mailing List Archive: Python: Python
finding most common elements between thousands of multiple arrays.

Index | Next | Previous | View Flat

hanooter at gmail

Jul 4, 2009, 12:33 AM

Views: 783
Permalink

finding most common elements between thousands of multiple arrays.

Currently I need to find the most common elements in thousands of
arrays within one large array (arround 2 million instances with ~70k
unique elements)

so I set up a dictionary to handle the counting so when I am
iterating I up the count on the corrosponding dictionary element. I
then iterate through the dictionary and find the 25 most common
elements.

the elements are initially held in a array within an array. so I am am
just trying to find the most common elements between all the arrays
contained in one large array.
my current code looks something like this:
d = {}
for arr in my_array:
-----for i in arr:
#elements are numpy integers and thus are not accepted as dictionary
keys
-----------d[int(i)]=d.get(int(i),0)+1

then I filter things down. but with my algorithm that only takes about
1 sec so I dont need to show it here since that isnt the problem.

But there has to be something better. I have to do this many many
times and it seems silly to iterate through 2 million things just to
get 25. The element IDs are integers and are currently being held in
numpy arrays in a larger array. this ID is what makes up the key to
the dictionary.

It currently takes about 5 seconds to accomplish this with my current
algorithm.

So does anyone know the best solution or algorithm? I think the trick
lies in matrix intersections but I do not know.
--
http://mail.python.org/mailman/listinfo/python-list

Subject	User	Time
finding most common elements between thousands of multiple arrays.	hanooter at gmail	Jul 4, 2009, 12:33 AM
Re: finding most common elements between thousands of multiple arrays.	clp2 at rebertia	Jul 4, 2009, 12:45 AM
Re: finding most common elements between thousands of multiple arrays.	andreengels at gmail	Jul 4, 2009, 1:04 AM
Re: finding most common elements between thousands of multiple arrays.	vilya.harvey at gmail	Jul 4, 2009, 2:55 AM
Re: finding most common elements between thousands of multiple arrays.	neilcrighton at gmail	Jul 4, 2009, 6:39 AM
Re: finding most common elements between thousands of multiple arrays.	steve at REMOVE-THIS-cybersource	Jul 4, 2009, 6:42 AM
Re: finding most common elements between thousands of multiple arrays.	steve at REMOVE-THIS-cybersource	Jul 4, 2009, 7:38 AM
Re: finding most common elements between thousands of multiple arrays.	hanooter at gmail	Jul 4, 2009, 9:22 AM
Re: finding most common elements between thousands of multiple arrays.	vilya.harvey at gmail	Jul 4, 2009, 10:59 AM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 4, 2009, 7:19 AM
Re: finding most common elements between thousands of multiple arrays.	steve at REMOVE-THIS-cybersource	Jul 4, 2009, 6:39 PM
Re: finding most common elements between thousands of multiple arrays.	python at rcn	Jul 8, 2009, 12:11 AM
Re: finding most common elements between thousands of multiple arrays.	lie.1296 at gmail	Jul 4, 2009, 11:36 AM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 4, 2009, 12:51 PM
Re: finding most common elements between thousands of multiple arrays.	hanooter at gmail	Jul 4, 2009, 3:06 PM
Re: finding most common elements between thousands of multiple arrays.	python at mrabarnett	Jul 4, 2009, 3:29 PM
Re: finding most common elements between thousands of multiple arrays.	hanooter at gmail	Jul 4, 2009, 3:33 PM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 4, 2009, 3:50 PM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 5, 2009, 5:30 PM
Re: finding most common elements between thousands of multiple arrays.	steve at REMOVE-THIS-cybersource	Jul 5, 2009, 8:28 PM
Re: finding most common elements between thousands of multiple arrays.	__peter__ at web	Jul 5, 2009, 11:33 PM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 6, 2009, 6:59 AM
Re: finding most common elements between thousands of multiple arrays.	python at mrabarnett	Jul 4, 2009, 4:05 PM
Re: finding most common elements between thousands of multiple arrays.	emile at fenx	Jul 4, 2009, 6:00 PM
Re: finding most common elements between thousands of multiple arrays.	steve at REMOVE-THIS-cybersource	Jul 4, 2009, 6:31 PM
Re: finding most common elements between thousands of multiple arrays.	andrew.henshaw at gtri	Jul 7, 2009, 2:48 PM
Re: finding most common elements between thousands of multiple arrays.	Scott.Daniels at Acm	Jul 10, 2009, 1:38 PM

Index | Next | Previous | View Flat

Interested in having your list archived? Contact lists@gossamer-threads.com