Let me set the stage.
In late 2018, CDX was in a bad place. We had been hit by malware a while back, which highlighted a security hole in our system. Our image gallery software, Gallery2, had been abandoned by developers, and there were no patches or updates in sight. This connection to Gallery 2 from the main website was also keeping us from upgrading the main website, because there was no connector available for newer versions of the website. So it was decided then, that In order to close the security hole, and to be able to upgrade the core site, that Gallery2 had to go.
At the same time, revenue had been steadily dropping. Most of our revenue came from YouTube and paid direct advertising, but that had begun to dry up. Our stable of contributors had left to do their own thing, and without them, it was up to one person to create the content that brought in advertisers, and that person was tired and could not keep up.
In order to be fair to our advertisers, I either canceled them or opted not to renew their contract. I felt that if we weren’t making the effort to bring traffic to them, it was not fair to take their money. And YouTube has been eating itself for the last few years, so that revenue had slowed to a trickle.
The site costs were $270 a month for our dedicated server. There once was a time when advertising covered that, but not any more. I had to really take a hard look at the situation. Did I want to invest the money to do all the migration and updates? Did I want to continue making what was basically an extra car payment out of my own pocket each month? I got one estimate of the migration at 10k!
I decided to put it out there to the fans. I thought, maybe if it meant enough to them, they would pitch in for the migration. I set the goal at around 3k. I thought, if I didn’t meet the goal, then that would be a decider.
In the meantime, I reached out to my web host, and asked them what we could do to reduce my hosting cost. They said my machine was the cheapest dedicated server they had, but they had an option for me - Cloud hosting. We went through the numbers and it looked as if it would save me about a hundred bucks a month. So I thought, if the fundraiser met its goal, maybe it could pay for the new hosting for the year while I sorted out what was going on with Gallery2.
The fundraiser started strong, with lots of die-hard fans pitching in. It meant so much to me in the beginning days that it renewed my commitment to keeping the site alive.
We migrated to Azure in April and it cost me $700 in labor to move it. The site clocked in at around 600GB, which is very large. We ended up purchasing 1TB of storage so that should have handled it.
Well, it wasn’t the storage that was the issue, it was the processing power of the Virtual Machine (VM). It just wasn’t powerful enough and the site crashed immediately. Once I upgraded the VM the site ran like a purring kitten. But I knew that would increase the cost.
I then began cleaning out the site. I backed up all of the podcasts. All of the Destroy All Podcasts DX shows, all of the VEEF shows, even the short lived PowerToku shows. Got them all off the server.
Then the first bill came. At nearly $350, it was MORE than the original hosting cost! How could this happen? I was devastated. When I got access to the billing portal it appeared that half of my cost was storage. My understanding at the time was due to the size of the site, we were being billed more for the bandwidth.
It was then that I resolved to dig my heels in. This was early summer 2019 at this point.
Now to understand what I had to do next, it requires a little explanation of how CDX works.
We had two systems that worked together. The content management tool (CMS) is Drupal, and the image management tool was Gallery2. To add images to a review, you would upload them to a gallery in Gallery 2, where each image had a code like [G2:12345]. When writing the review we would then insert that code in the body where we wanted that image to appear. A connector would look at the review, see the code, and say “get me this image and display it in the body”.
Well, in order to remove that gallery, I had to remove that code from every review, article, and news item. Thousands of pieces of content. Then I had to replace them with raw HTML that linked directly to the image on the file system.
There was no way I could do this all by hand, all by myself, so I brought in a programmer to build a drupal module that would run a script that converted each instance automatically. I paid a few hundred dollars for that, and it worked, sort of. The big flaw was I could not batch process them with the module, meaning, I had to do them one at a time.
So I did.
Also at this time the fundraiser had begun to fizzle out, only reaching ⅔ of its goal.
I went through every single piece of content on the site by hand. On a good day I could get about 100 done. It consumed my entire summer. And not only did I fix the images, but I also fixed broken links, fixed videos that had broken embed codes (due to our switch from http to https years earlier). I finished in October, all the while getting $300+ server bills. I think I processed something like 6,000 pieces of content.
I also discovered about 400GB of temporary files that Gallery was supposed to be deleting regularly but never did. After all that work, our 600GB monster trimmed down to a svelte 26GB.
But what did not go down was the cost. This whole time I misunderstood how the pricing for virtual machine storage worked. You are billed on what you have allocated, not what you use. So I was paying for the entire 1TB, even though I was using only 25GB.
I was able to downgrade the VM, so that saved a little money.
The next step was to reduce the disc size. I thought that we could just resize it, but no, I have to install a new disc, migrate the data, blah blah blah to the tune of $150 an hour. Another setback.
I decided to reach out to the CDX audience to see if anyone knew anything about Azure, and could help out. One guy did, a fan named Alvaro.
Once Alvaro got into the VM, he made a discovery that explained why I was paying so much. Apparently we were being billed for SIX hard drives instead of the two we needed. They were created somehow during the migration process back in April, and I have been charged for them ever since.
I would like to say I could blame the hosting company for never explaining my breakdown of costs like I asked. Or the guy who did my migration. But in reality, I was in over my head and I should have known better. I should have learned more about Azure how it worked, and if I knew more about the Azure portal, I would have caught those extra drives earlier. But i was so deep in the process of converting the content over that I didn’t even think that was the problem at all.
There were a few realizations there.
- If I had known about the 400GB temp file, I may not have had to do the 1TB drive in the first place. I could have gone with a much smaller drive.
- If I knew about the extra drives, I could have saved over $200 a month in hosting fees.
- I need to learn more about Azure and cloud hosting in general.
- My host sucks, at least in giving you the information you need to make educated decisions.
So here we are now, in November of 2019.
- All the extra drives are removed.
- All the data has been migrated to a smaller drive.
- The VM has been resized.
What’s next? The site will continue to exist, and I’ll update it when I feel like it. I’ll still welcome contributors old and new, but I'm no longer going to feel like I have an obligation to any sponsor or platform. Even just this week, YouTube changed their policies so that I had to manually review over 2,000 videos and set them to “for kids” or “not for kids”.
I’m buying a house, so the winter and spring are going to be dedicated to settling in and beginning a new life there. In the spring, we may look at upgrading our CMS to a newer version, but for now, I need a break.
Anyway, thanks for reading, and thanks for being a fan of CollectionDX. Your donations, patience and kind words this year have meant the world to me, and it reminded me that CollectionDX really belongs to all of us, not just me, and it deserves to be preserved as long as it can be.
And I want to extend a special thanks to Alvaro, who stepped in and helped us cross the finish line.
Thanks
JoshB
