The Ryan Hellyer AWS Nightmare


(Leland Fiegel) #1

Read this if you haven’t already: http://wptavern.com/ryan-hellyers-aws-nightmare-leaked-access-keys-result-in-a-6000-bill-overnight

Short version

  • Ryan Hellyer, a well-known WordPress developer, open sources his website code (like many people do)
  • Ryan was sure to remove his wp-config.php file from the repo, because it contained his AWS key, among other critical items.
  • A file called wp-config.php.save was mistakenly left in the repo, which contained the exact same stuff wp-config.php did
  • Someone finds the AWS key and opens up a bunch of servers on Ryan’s account, eventually racking up a bill of almost $6,000.
  • Much of the damage was done within 12 hours of opening the Git repository.

My Thoughts

It’s scary that this process seems to be so automated. I’d imagine there are bots that scan open source repos for exposed AWS access keys just like this. And once a “hot” one is found, go to town on creating a bunch of servers, probably for spam or some other nefarious purpose.

At the time Amazon sent out its “suspicious activity” email, they should’ve just suspended the account from making any further instances. I’m not sure why they allowed it to continue at that point.

I’m a big believer in learning from mistakes, even if they turn out to be expensive. But $6,000 for something like this would be a tough pill to swallow.

Hopefully Amazon grants the concession for unauthorized activity. I’ve never heard of something like this happening before, but he can’t be the only one.

After that, maybe Amazon should set up some honeypot to catch the people that illegally rack up thousands of dollars in hosting bills on other people’s dimes.

If you use AWS to run your technology stack, even if you don’t open source your code, it may be a good idea to read up on access policies which could prevent something like this.


(Ryan Hellyer) #2

Thanks for sharing this here.

It certainly wasn’t the most pleasant experience here, but I think it will be okay in the end. It sounds like Amazon routinely provides concessions for stuff like this, so hopefully that is the case for me.


(Leland Fiegel) #3

That would be great to hear. Definitely keep us updated.


(Jeff C) #4

Yeah, let us know if they went ahead with the concession. Do you have 6 grand in the couch cushions just to be safe?


(Ryan Hellyer) #5

Yep, the concession was granted :smile:

I could have covered the bill if I absolutely had to, but I certainly wouldn’t have been happy about it!


(Stephen Cronin) #6

Congrats Ryan. That must be a relief! So glad they did the right thing.


(Leland Fiegel) #7

That’s awesome to hear! :smiley:


(James Huff) #8

Hey! This is WPChat, not WPBrag! j/k glad to here they were accommodating! :slight_smile: