Recent PFC Website Outage

Dear Readers and Subscribers,

We understand many of you have been wondering about our recent downtime over the past few days. Here’s what happened and how we’ve been working to resolve the issue.

The root cause of the outage was a major infrastructure failure at one of our hosting providers—a well-known company. Their edge firewall completely broke down, leaving our server inaccessible and unable to connect to the internet. After two days of waiting for a resolution, it became clear that no immediate fix was forthcoming. At that point, we made the decision to cut our losses and rebuild our systems from backups on a new server with a different provider.

The Recovery Process

Initially, I secured a temporary server using one of the few hosting accounts we could access without requiring two-factor authentication (2FA) via email. However, the assigned IP range for that server was 66.xx.xx.66/24, with a gateway address of 66.xx.xx.65. For obvious reasons, I rejected this server and continued searching for a more suitable option.

This process was further complicated by the fact that our organization’s email system was also down, making it impossible to verify 2FA emails for existing or new hosting accounts. After exhausting the options available within the organization, I took the initiative to procure a server through my own business. Unfortunately, my first two orders were rejected without explanation, wasting another day as I waited for the company to cancel the orders.

Finally, I was able to secure a dedicated server from another provider. Within 8 minutes of placing the order, I had access to the new server and began the recovery process late last night. This involved configuring the new server, establishing it within the cluster, restoring data from backups, transitioning our database from MySQL 8 to MariaDB 11, re-tuning the database, and preparing the system for operation.

Current Status

We’ve successfully restored most of our websites, but approximately 5 hours of data—including emails, posts, and other content—has been lost due to the need to recover from backups. Unfortunately, the original server remains inaccessible, but once it becomes available, I’ll work to recover any missing data, including emails and database fragments.

There’s still a lot of work ahead, including resetting email passwords for hundreds of accounts, fully restoring backend tools for our volunteers, and ensuring all systems are stable. In total, we’re recovering over 600GB of data.

Next Steps

This new server is a temporary solution to get us back online quickly. Later this month, we’ll likely schedule additional downtime (6–12 hours) to transition to a dedicated server fully under our organization’s control. We’ll provide clear notifications at the time to minimize disruption.

Lessons Learned and Improvements

We’re taking several steps to improve our infrastructure and reduce the risk of future outages:

• Email Server Separation: We’ll move our email system to a separate server to ensure it remains operational during any future crises.
• Faster Backup Windows: We’ll reduce our backup window from the current 6 hours to 1–3 hours, minimizing potential data loss and recovery times.
• Increased Redundancy: We’ll explore over-capacity options on other servers in our cluster to handle emergencies, especially for our largest website, which exceeds 170GB.
• Improved Recovery Times: With faster backups and spare capacity, recovery times will be reduced to just a few hours in the future. When the decision to recover over waiting out a technical issue is made.

What Worked Well

Despite the challenges, our cluster technology and backup systems performed well. DNS records updated automatically and propagated quickly, enabling a faster recovery process. While the database transition required some manual intervention, the overall system design proved resilient.

We sincerely apologize for the inconvenience caused by this outage. Our editors are already preparing new posts, and our newsletter will resume tomorrow, January 27, 2025. Thank you for your patience and understanding as we work to improve our systems and ensure greater reliability moving forward.

Kind regards,

PFC-IT

Your Tax Free Donations Are Appreciated and Help Fund our Volunteer Website

Disclaimer: We at Prepare for Change (PFC) bring you information that is not offered by the mainstream news, and therefore may seem controversial. The opinions, views, statements, and/or information we present are not necessarily promoted, endorsed, espoused, or agreed to by Prepare for Change, its leadership Council, members, those who work with PFC, or those who read its content. However, they are hopefully provocative. Please use discernment! Use logical thinking, your own intuition and your own connection with Source, Spirit and Natural Laws to help you determine what is true and what is not. By sharing information and seeding dialogue, it is our goal to raise consciousness and awareness of higher truths to free us from enslavement of the matrix in this material realm.