A Paypal Security Hole and Poor Customer Service Judgement that Made it Worse
I have been having problems for a while receiving Paypal payments to my business account. Today, I received an account notification for someone else's paypal account. I have received phishing and spoof emails before, but I was pretty sure this one was legit. I contacted the other person whose account notification I had received (they were horrified at that security breech, by the way), and sure enough, they were honest enough to admit they had been receiving some mystery payments they could not account for, which we quickly determined were mine. I asked them to check their email addresses on their account, and sure enough, for some reason neither of us could fathom, my email address was listed as a secondary address on their account. This is the same email that is the primary on my Paypal account, something Paypal claims is impossible.
I asked the other user to not touch it for a minute, and said I wanted to try an experiment. I called Paypal and got a real person (a slog in and of itself) and described the situation: I had solid reason to suspect that my email address on my account was on someone else's account as well. They said that was impossible. I insisted it might be possible. Eventually, the customer service agent relented and said they would run a search (I presume they search their data base for my email address and check for multiple hits, an assumption later confirmed by the supervisor).
Well, the customer service agent returned and said "I am happy to tell you your account is fine and no one else has your email address." She actually said the "happy" thing in a chirpy voice. I said that now I was REALLY worried, as I had definitive evidence my email is on another account, and if their search programs are not finding the issue, I have no confidence that it is not on more accounts. After getting nowhere with this, I asked for a supervisor.
I explained all of the above, and the supervisor admitted the first agent did not tell me the whole truth. She said, "yes, in fact we did find your email on one other account and eliminated it. The problem was on just that one other account. We have had this problem a few times and are still trying to figure out why it happens because it should be impossible." Fine. But why did the customer service agent feel the need to lie? I guess technically it was correct for her to report that my email was not on any other account, as they had eliminated the duplications before they took me off hold. It just seems to be in the institutional nature of organizations to cover their errors and not admit them.
I guess this sort of thing might work with the average computer user who is unsure of his skills and can be convinced that he misunderstands the problem. And to be fair, all of computer and software customer service seems to work this way, trying to convince users it was their error rather than a bug. But in my case, knowing for an absolute fact that there was an error, this approach only panicked me more, as I became worried not only with the security hole in their payments system, but with the fact that the company was apparently unaware of the hole and unable to detect it.
The other issue is that I actually think I know how this happened, but neither the agent nor their supervisor took the time to try to get any background information on me that might help them diagnose what is obviously a bug in their system they have been chasing unsuccesfully. It is a bit like having a mystery epidemic where a disease is spreading via an unknown vector but no one is doing any research into the patients' histories. Yeah, I know they can't put a priority on every bug fix, but I would assume that for a payments processor a bug that allows money to flow to the wrong person might be of some priority.
Postscript: Not that it matters to any of you, but here is my hypothesis. I actually had done a transaction with this other user years ago. This user did not have a paypal account at that time, but one can actually send money via credit card to someone with a Paypal account even if the person sending money does not have an account. The other user sent me the money with her Visa card from a public terminal, but called me because she could not complete the form because she did not have an email address. I told her just to plug mine in, and if I got any emails on the transaction I would mail them to her. Years later, she was more sophisticated and opened up her own Paypal account. My hypothesis (really, the only explanation that works) is that at the time she signed up, the Paypal computer went back into its records, found her name from this old transaction, and automatically attached the old email address (mine) from that transaction to the new account as an additional email. Since this email was not entered via the data entry screen, it bypassed the duplicate email name check which presumably happens at data entry. It is a back door that allows duplicates in. I strikes me someone intheir development group might be interested in this hypothesis, since this is one of those bugs it is hard to track down, but no one asked.
Jens Fiederer:
Hopefully the right programmer, or a friend of theirs, reads your blog!
Speaking as a programmer myself, this sort of data can be SO helpful to diagnose a problem. If you read news about a PayPal customer service representative being strangled, we have a suspect.
January 14, 2009, 10:31 amDave:
The obvious answer as to why customer support operates in the manner you describe is that companies view it as their job to do anything and everything to avoid liability.
For the same reason doctors faced with irate patients who threaten to sue are told to shut their mouths and ignore their patients' complaints.
Blame the malignant influence of lawyers on society.
January 14, 2009, 10:39 amDave:
I would also argue, aside from the obvious lawyer issue I describe above, that most, if not all, customer support employees (supervisor or non-) don't really understand how databases work or how the software that interacts with the database is designed.
They "know" from experience that "X can't happen; therefore, the customer is in error," ignoring the possibility that "X can happen if this unusual set of circumstances apply." There is a whole level of nuance and sophistication that you bring to bear in your post that is beyond the purview of people working in customer support functions.
January 14, 2009, 10:43 amcolson:
I read the blog regularly ;). I'm just not the right engineer! Doh!
I'm sorry to hear about the problems with our customer support team and our system. For your information and contrary to what you were told, we do have a bug filed with our engineers on the issue and it is being actively worked on. From the details that I have, available on the issue - it appears to be exactly the problem you mentioned in your Postscript. If you would like, you can shoot me an email Mr. Coyote.
@Dave - I don't really disagree with your assertion that 'They "know" from experience...' at all. Your insight is almost the basic premise of troubleshooting anything related with software which, as you note, does not always work.
January 14, 2009, 4:20 pmtxjim:
The upside: I think you have provided the key pieces of info and I think your diagnosis is on target. The downside, as you discovered, is this is absolutely not the first time this has happened.
January 14, 2009, 7:52 pmThomas:
I know someone who is a programmer with Paypal, I sent him the link. No promises of course, but I tried.
January 15, 2009, 8:48 amEd (Buildaskill.com):
Hi Coyote
I'll plug this post in this week's edition of Sunday Papers in the 'money' section - I know we have several eBay and PayPal staff who are readers - the more this gets circulated, the more effort they will apply to get the loophole closed.
It's certainly a worry - thanks for a great post
Ed
January 15, 2009, 10:47 amdreamin:
Sounds to me like it's a known issue.
January 16, 2009, 5:06 amAllen:
As a software engineer what annoys me the most about this is that the claim of "impossible" is absolutely false. They're storing data in a database. That database has no concept of any of the business rules. In this case it's probably something like an email address can not belong to more than one account. Now the front end code could account for that. The problem is companies do all sorts of things with the data beyond the front end application. They upgrade their database, they transfer data from one application to another, et al. Those processes too must enforce that rule. On top of it, it would appear that they don't any a simple quality assurance check in place to double check existing data to ensure it conforms to these rules. Since humans are the ones telling the database what to do, it is actually impossible for it to be impossible for this sort of thing to happen. Humans make mistakes. The more data and more complex the systems and the more data moving between them, the more likely some human is going to make at least one mistake.
If you pursue this any further, I recommend letting them know that impossible is a false claim and ask them to describe to you what sort of QA things they do. Of course they won't and shouldn't give you low level details on those but the point is to get them to admit things could go wrong and try to show they do things to prevent that and even fix it in the event it does occur.
January 16, 2009, 10:35 amWeronika:
Colson - it's good to know someone at PayPal is willing to "own the problem" and take his role as a representative of the company seriously. But I think there just aren't enough employees like you to deal with what's going on there right now.
More than a week ago someone gained access to my BF's eBay/PayPal account and ordered a $1200 laptop sent to an unconfirmed address. My BF filed meticulously documented reports with eBay, PayPal, and all the police and gov't agencies that deal with identity theft and mail fraud... Several phone calls to PayPal later, random customer service people keep telling him that the case is "under review." Not very reassuring, and not what you'd expect from the company that stakes its reputation on security of transactions.
January 17, 2009, 1:57 pmlinklover:
What is being missed here is that in big companies there is big distance between software developers and customer support. Developers develop products, tell "what they think they need to tell" CS teams, and off the CS teams go, "supporting" the product.
When problems get reported by users, they are sent back to developers only when things are happening at sufficient volumes.
Today's software and hardware products are unbelievably complex, compared to, say, VCRs or cameras from 20 years ago; and they change at fast pace. There is no way this is the fault of CS teams.
January 18, 2009, 9:43 pmH.L. Hill:
Thanks Coyote, and others.
I was just getting ready to add pay-pal to my business. I don't think I can afford to do so at this time. I don't need these kinds of problems with a start-up business. I am fairly new and inexperienced with this form of internet use, so this information is extreeemly important to me. Thanks again gentlemen, and ladies too!
H.L. Hill
January 19, 2009, 7:38 amPresident and Founder
David C:
I am also disappointed at Paypal's customer service. Here is why:
Recently, our business paypal account has received a bunch of high dollar amount payments for product purchase on our web site. These payments were determined Unauthorized" later on and so we lost thousands of dollars since we already shipped the goods and all packages were delivered.
The horrible side of this story is that this kind of payments keep coming in and I called their customer support and want them to further investigate it. What the agent told me was that we need to confirm the order (payer information) using whatever available internet white pages to manually verify the that the account owner (paypal account/credit card owner) indeed made the purchases!
If I keep accepting paypal, I will lose more money. Now I am thinking very hard on other paypal alternatives such as goolge check out or checkout.com.
My suggestion to HL Hill: you made the right decision not to take paypal payment!
February 7, 2009, 5:00 am