Mailing List Archive: ModPerl: ModPerl

Any success with storing photos in a database?

1 2

View All

Index | Next | Previous | View Threaded

mpeters at plusthree

Oct 15, 2008, 10:51 AM

Post #26 of 34 (806 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

Mark Stosberg wrote:

> At one point in the past I did a variation of this where we put the next ID of
> a related database sequence in the form, and this would become the new primary
> key when inserted, and it would of course not allow the same primary key to be
> used twice. That worked, but I realized was open to abuse if a user tweaked the
> number to be larger than the sequence. Then, eventually a legitimate user would
> eventually be assigned that value by the sequence, and it would fail.
>
> Are their specific modules that you recommend to help with this?

You can try a UUID (look at Data::UUID). It's not sequential, but someone could still tweak the form
to create a value that could potentially conflict with another value in the future. But I'd say it's
much less likely than a sequential id.

--
Michael Peters
Plus Three, LP

simon-lists at ldml

Oct 15, 2008, 11:02 AM

Post #27 of 34 (812 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

On 15 Oct 2008, at 18:49, Mark Stosberg wrote:

> On Wed, 15 Oct 2008 12:41:55 -0400
> "Perrin Harkins" <perrin [at] elem> wrote:
>
>> On Wed, Oct 15, 2008 at 12:31 PM, Mark Stosberg
>> <mark [at] summersault> wrote:
>>> We had a "double submit" bug that allowed a form to be submitted
>>> twice when we
>>> weren't fully prepared for that. We are still researching the best
>>> practices to
>>> address this a general case. One approach we are considering is
>>> change the
>>> submit action on forms with JavaScript, so it disables the submit
>>> button, and
>>> then actually submit the form, preventing one kind of double-
>>> submission. It
>>> seems like I don't see this approach happening in the wild much,
>>> though. I
>>> suspect there is a better solution.
>>
>> JavaScript is okay, but can be a problem when people hit back
>> expecting to use the form again and the button is still disabled.
>
> Thanks for the response.
>
> That's what I was concerned about. I don't have a sense of how
> much this would happen in practice.

Take some elements from the form and search for an add in the past
<time period>. We did one recently where we took the email address,
recipient and amount from the form and then made sure that the same
data hadn't already been submitted in the last 24 hours - working on
the principle that a user is unlikely to gift the exact same amount to
the same person on the same day. The check is trivial compared to the
commit.

FWIW, we looked into this and this approach seemed to be the quickest
and most reliable available. If you Google on this, you may find an
article or two which support this approach.

HTH

Simon

cosimo at streppone

Oct 15, 2008, 11:11 AM

Post #28 of 34 (810 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

On Wed, Oct 15, 2008 at 12:31 PM, Mark wrote:

> We had a "double submit" bug that allowed a form to be submitted twice
> when we weren't fully prepared for that. [...]
> One approach we are considering is change the submit action on forms
> with JavaScript, so it disables the submit button

That's the cheapest option IMO, and it works very well
for 95% of the typical double submit cases (like Windows
users with the "double-click" syndrome :).

On submit event I checked for a flag to be "1".
If "1", then form has been already submitted.
If "0", set to "1" and submit.

Then Perrin wrote:

> JavaScript is okay, but can be a problem when people hit back
> expecting to use the form again and the button is still disabled.

You could always trigger some javascript event on page load
the re-enables the submit control, even if already enabled,
resetting the flag to "0". This is what I did.

Then, if you want to be 100% sure, you have to work
on the server side...

--
Cosimo

mark at summersault

Oct 15, 2008, 12:20 PM

Post #29 of 34 (811 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

On Wed, 15 Oct 2008 12:41:55 -0400
"Perrin Harkins" <perrin [at] elem> wrote:

> On Wed, Oct 15, 2008 at 12:31 PM, Mark Stosberg <mark [at] summersault> wrote:
> > We had a "double submit" bug that allowed a form to be submitted twice when we
> > weren't fully prepared for that. We are still researching the best practices to
> > address this a general case. One approach we are considering is change the
> > submit action on forms with JavaScript, so it disables the submit button, and
> > then actually submit the form, preventing one kind of double-submission. It
> > seems like I don't see this approach happening in the wild much, though. I
> > suspect there is a better solution.
>
> JavaScript is okay, but can be a problem when people hit back
> expecting to use the form again and the button is still disabled.

Thanks for the response.

That's what I was concerned about. I don't have a sense of how
much this would happen in practice.

> Another approach is a unique ID in the form that you track in the
> user's session (i.e. this ID was seen before). If the problem is
> large uploads with no feedback until they finish, you can use one of
> the upload progress tools.

At one point in the past I did a variation of this where we put the next ID of
a related database sequence in the form, and this would become the new primary
key when inserted, and it would of course not allow the same primary key to be
used twice. That worked, but I realized was open to abuse if a user tweaked the
number to be larger than the sequence. Then, eventually a legitimate user would
eventually be assigned that value by the sequence, and it would fail.

Are their specific modules that you recommend to help with this?

Mark

mark at summersault

Oct 15, 2008, 12:20 PM

Post #30 of 34 (802 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

> Take some elements from the form and search for an add in the past
> <time period>. We did one recently where we took the email address,
> recipient and amount from the form and then made sure that the same
> data hadn't already been submitted in the last 24 hours - working on
> the principle that a user is unlikely to gift the exact same amount to
> the same person on the same day. The check is trivial compared to the
> commit.
>
> FWIW, we looked into this and this approach seemed to be the quickest
> and most reliable available. If you Google on this, you may find an
> article or two which support this approach.

This does seem workable and reliable. I suppose I'm hoping for an
approach that can easy re-used across many forms, without much per-form
effort required.

Also, I think for some kinds of changes being submitted, the check of
"has this happened recently" won't apply.

For example, we allow people to re-order a list of photos. They may well
reverse the ordering a few seconds later and put the photos back in the
original order. (Although that example may not be vurnerable to a
double-submit bug, as it would be valid to submit the same ordering
request twice. )

Mark

mark at summersault

Oct 15, 2008, 12:20 PM

Post #31 of 34 (813 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

On Wed, 15 Oct 2008 13:51:21 -0400
Michael Peters <mpeters [at] plusthree> wrote:

> Mark Stosberg wrote:
>
> > At one point in the past I did a variation of this where we put the next ID of
> > a related database sequence in the form, and this would become the new primary
> > key when inserted, and it would of course not allow the same primary key to be
> > used twice. That worked, but I realized was open to abuse if a user tweaked the
> > number to be larger than the sequence. Then, eventually a legitimate user would
> > eventually be assigned that value by the sequence, and it would fail.
> >
> > Are their specific modules that you recommend to help with this?
>
> You can try a UUID (look at Data::UUID). It's not sequential, but someone
> could still tweak the form to create a value that could potentially conflict
> with another value in the future. But I'd say it's much less likely than a
> sequential id.

So how might an implementation look?

We could create table of "form_uuids". Each time a form is submitted, we check
if the UUID we have is has already been used.

If not, we insert the new UUID and proceed.

Every so often, the table could cleaned up via cron, (since we probably don't
care about seeing the same UUID weeks apart, just seconds or minutes apart).

There is still room for a small race condition in between checking to see if we
used the UUID and inserting it, but I think that may be acceptable.

Mark

mpeters at plusthree

Oct 15, 2008, 12:46 PM

Post #32 of 34 (805 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

Mark Stosberg wrote:

> So how might an implementation look?

I would either make the uuid the primary key (might affect performance since it's not an integer,
but a string) or a unique key for the same table. Then you don't have anything else to keep track of
(no extra tables, etc).

> Every so often, the table could cleaned up via cron, (since we probably don't
> care about seeing the same UUID weeks apart, just seconds or minutes apart).

UUID's should never collide.

> There is still room for a small race condition in between checking to see if we
> used the UUID and inserting it, but I think that may be acceptable.

If you're really worried about someone attacking you in this way then insert the record with the
uuid first and then let them upload. If you don't find the uuid they are trying to upload to, then
they changed it so just disallow the upload.

--
Michael Peters
Plus Three, LP

eric at pieinsky

Oct 15, 2008, 8:45 PM

Post #33 of 34 (798 views)
Permalink

Re: Any success with storing photos in a database? (prevents double-submits) [In reply to]

On Oct 15, 2008, at 11:11, Cosimo Streppone wrote:

> On Wed, Oct 15, 2008 at 12:31 PM, Mark wrote:
[...]
> Then Perrin wrote:
>
>> JavaScript is okay, but can be a problem when people hit back
>> expecting to use the form again and the button is still disabled.
>
> You could always trigger some javascript event on page load
> the re-enables the submit control, even if already enabled,
> resetting the flag to "0". This is what I did.

I do it this way (via jQuery) and it works okay even when faced with
the browser's back-button:

function returnFalse() {
return false;
}
$(document).ready(function() {
// Take care of back-button issues first.
$('form').find(':submit').unbind('click', returnFalse);
// Then wire up the auto-disabling stuff.
$('form').submit(function() {
$(this).find(':submit').click(returnFalse);
return true;
});
});

I don't use the 'disabled' attribute as that prevents the browser (at
least some of them) from sending the button back to the server with
the rest of the form.

Eric Howe
eric [at] pieinsky

jonathan at 2xlp

Nov 24, 2008, 1:40 PM

Post #34 of 34 (627 views)
Permalink

Re: Any success with storing photos in a database? [In reply to]

chiming in a few months late...

from my experience, and responding to some thoughts in the thread:

- storing photos in mysql/pgsql is not a good idea. aside from misc
issues that always arise, you end up stressing the db through searches
- the better way would be to store photo meta-data in the system,
and then search for that to serve. its a much leaner search and
faster/more lightweight.

-we played around with a lot of things

-- mogilefs
--- i think flickr also used pre-yahoo. someone mentioned a
flickrbook, there is also a powerpoint from about 1year after they
launched

-- directory hashing
--- create an md5 and store in db; store on filesystem as ab/cd/ef/
gh/abcdefg + suffix ( [original,-thumb ,-medium].jpg )
---- use rewrite rules on nginx to loadbalance across cluster by
bucket and serve

-- i didn't like perlbal, but it can do some neat in-time
bitshifting on gifs to change the color palatte

-- the BEST solution we found and upgraded to was.. Amazon s3 +
Panther Express. this is before amazon offered cdn services
--- store files on amazon -- originals in private bucket, copies in
public buckets
--- serve files from panther, which was a kickass cdn. they cache
everything in their network , which offsets instabilities from amazon
--- don't worry about disk usage, clustering, load balancing,
BACKUP, and miscellaneous stuff designed around images
--- migrating to that setup took us 2 days... and it was a better
performer than 3 months of code and constant maintenance

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads