Scala Studios Status - Incident history https://scala.instatus.com Scala Studios Tue, 25 Feb 2025 15:00:00 +0000 Onsite DDoS Protection Development - More Progress Type: Maintenance Affected Components: CH-NODE-1, CH-NODE-2, CHI-D-01, CH-VM-01, CH-VM-02 Feb 25, 19:00:00 GMT+0 - Completed - Maintenance has completed successfully Feb 25, 22:46:10 GMT+0 - Identified - We have **completed maintenance for today**, this extension saw no external loss to production switches. We have made **a lot of real, meaningful progress today**. We are around the home stretch now, simply a few things to tidy up before we push **more live traffic** to onsite! We will be **continuing our work** again tomorrow at **9:00 AM Central** to avoid any potential interruptions during peak hours. Feb 25, 15:00:00 GMT+0 - Identified - At specifically **9AM Central Time** we will be continuing development with our onsite network. We apologize for any interruptions this may cause, we are doing this early in the AM to ensure we impact as little traffic as possible. We understand the frequent issues from upstream isn't making this an easy pill to swallow, but we must continue work on this to ensure we are independent from future network issues once Onsite goes live. To give context on what onsite will bring for those who may have missed it: Onsite DDoS Protection will allow us to have a **localized Path environment** that will be **isolated** from the **global network**. This will allow us to attempt to seamlessly, **without dropping traffic** move between the global network and local network when needed. Take for example the issues that we have been facing now, these are all from the **global network** with Onsite we would be-able to **push traffic onto our local mitigation stack**, effectively isolating us from global network issues, ensuring full uptime. **This also works in reverse**, if the local network was to go offline for some reason, we would route back onto the global network, while again attempting to not drop any traffic. This system is **extremely complex**, and has been under development for a while now, we are speeding up progress but also doing so at the least impactful times, we understand no downtime is ideal, however this must be done in order to ensure we have a fully redundant network. I am sure everyone is tired of these upstream issues, we are too, and we truly believe onsite is the solution we are all looking for. Feb 25, 21:07:25 GMT+0 - Identified - Hello everyone, apologies for the short notice but we will be extending the previous maintenance for a few more hours. Why? Because today we’ve made some crazy good progress, and we think onsite can realistically be done very soon. We everything is online we even have some live traffic being pushed to it. To iterate again, this will be the solution for us to attempt to seamlessly move traffic from the global upstream to our localized mitigation stack when they have issues. Effectively keeping uptime with as little to no drops as possible.Thank you for bearing with us, we will send out some cool pictures soon Feb 25, 15:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Affected Components: , , , ,

<p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>19:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>22:46:10</var> GMT+0</small><br><strong>Identified</strong> - We have **completed maintenance for today**, this extension saw no external loss to production switches. We have made **a lot of real, meaningful progress today**. We are around the home stretch now, simply a few things to tidy up before we push **more live traffic** to onsite! We will be **continuing our work** again tomorrow at **9:00 AM Central** to avoid any potential interruptions during peak hours..</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Identified</strong> - At specifically **9AM Central Time** we will be continuing development with our onsite network. We apologize for any interruptions this may cause, we are doing this early in the AM to ensure we impact as little traffic as possible. We understand the frequent issues from upstream isn't making this an easy pill to swallow, but we must continue work on this to ensure we are independent from future network issues once Onsite goes live. To give context on what onsite will bring for those who may have missed it: Onsite DDoS Protection will allow us to have a **localized Path environment** that will be **isolated** from the **global network**. This will allow us to attempt to seamlessly, **without dropping traffic** move between the global network and local network when needed. Take for example the issues that we have been facing now, these are all from the **global network** with Onsite we would be-able to **push traffic onto our local mitigation stack**, effectively isolating us from global network issues, ensuring full uptime. **This also works in reverse**, if the local network was to go offline for some reason, we would route back onto the global network, while again attempting to not drop any traffic. This system is **extremely complex**, and has been under development for a while now, we are speeding up progress but also doing so at the least impactful times, we understand no downtime is ideal, however this must be done in order to ensure we have a fully redundant network. I am sure everyone is tired of these upstream issues, we are too, and we truly believe onsite is the solution we are all looking for..</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>21:07:25</var> GMT+0</small><br><strong>Identified</strong> - Hello everyone, apologies for the short notice but we will be extending the previous maintenance for a few more hours. Why? Because today we’ve made some crazy good progress, and we think onsite can realistically be done very soon. We everything is online we even have some live traffic being pushed to it. To iterate again, this will be the solution for us to attempt to seamlessly move traffic from the global upstream to our localized mitigation stack when they have issues. Effectively keeping uptime with as little to no drops as possible.Thank you for bearing with us, we will send out some cool pictures soon.</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>15:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Tue, 25 Feb 2025 15:00:00 +0000 https://scala.instatus.com/maintenance/cm7klsftt00cea6alfxozsfof https://scala.instatus.com/maintenance/cm7klsftt00cea6alfxozsfof Path-Routed Services - Packet Loss Type: Incident Duration: 19 hours and 3 minutes Affected Components: Website, Jetbrains Hub Service, Jetbrains TeamCity Service, CHI-D-01, Jetbrains YouTrack Service, jFrog Artifactory Service, Grafana, Sentry Services, CH-VM-01, Pterodactyl Panel, CH-VM-02 Feb 20, 17:16:07 GMT+0 - Resolved - Path's Frankfurt is now fully online. Feb 19, 22:12:49 GMT+0 - Identified - We have confirmed that our upstream provider is experiencing heavy packet loss and they are investigating. Feb 19, 22:51:05 GMT+0 - Monitoring - We will be re-routing traffic to a more stable upstream very soon. Within the next 10 to 20 minutes you may notice a drop as we migrate live traffic onto a more reliable provider.Apologies once again for the issues, we are working swiftly to resolve all of this. Feb 20, 00:41:46 GMT+0 - Identified - Path is about to bring up their Miami PoP, they are waiting for their ISP. This will add extra capacity which is very much needed. We are also going to be load balancing our traffic between both providers shortly. Feb 20, 17:15:28 GMT+0 - Monitoring - We are currently moving over a Prefix in our London location to another provider, you may notice a slight drop Feb 20, 00:42:02 GMT+0 - Identified - We are currently moving over a Prefix in our London location to another provider, you may notice a slight drop. Feb 20, 17:15:52 GMT+0 - Monitoring - Path will be pushing Frankfurt back online soon. Traffic will rebalance once this PoP is placed into production. Type: Incident

Duration: 19 hours and 3 minutes

Affected Components: , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>17:16:07</var> GMT+0</small><br><strong>Resolved</strong> - Path's Frankfurt is now fully online..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>22:12:49</var> GMT+0</small><br><strong>Identified</strong> - We have confirmed that our upstream provider is experiencing heavy packet loss and they are investigating..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>22:51:05</var> GMT+0</small><br><strong>Monitoring</strong> - We will be re-routing traffic to a more stable upstream very soon. Within the next 10 to 20 minutes you may notice a drop as we migrate live traffic onto a more reliable provider.Apologies once again for the issues, we are working swiftly to resolve all of this..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>00:41:46</var> GMT+0</small><br><strong>Identified</strong> - Path is about to bring up their Miami PoP, they are waiting for their ISP. This will add extra capacity which is very much needed. We are also going to be load balancing our traffic between both providers shortly..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>17:15:28</var> GMT+0</small><br><strong>Monitoring</strong> - We are currently moving over a Prefix in our London location to another provider, you may notice a slight drop.</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>00:42:02</var> GMT+0</small><br><strong>Identified</strong> - We are currently moving over a Prefix in our London location to another provider, you may notice a slight drop..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>17:15:52</var> GMT+0</small><br><strong>Monitoring</strong> - Path will be pushing Frankfurt back online soon. Traffic will rebalance once this PoP is placed into production..</p> ]]> Wed, 19 Feb 2025 22:12:49 +0000 https://scala.instatus.com/incident/cm7cgyopz005c24stv5lbui2q https://scala.instatus.com/incident/cm7cgyopz005c24stv5lbui2q Partial Deployment of On-Site DDoS Protection Type: Maintenance Duration: 1 hour and 46 minutes Affected Components: Website, Jetbrains Hub Service, Jetbrains TeamCity Service, CHI-D-01, Jetbrains YouTrack Service, jFrog Artifactory Service, Grafana, Sentry Services, CH-VM-01, Pterodactyl Panel, CH-VM-02 Feb 19, 22:02:39 GMT+0 - Completed - Maintenance has completed successfully Feb 19, 20:14:33 GMT+0 - Identified - After the successful testing of our on-site DDoS protection into our network core in the last maintenance, we are moving onto stage 3, which is the beginning of moving production traffic to the protection appliances. We are keeping production traffic to a minimum to ensure no adverse effects in the initial rollout. Over the next few days, we will be shifting more traffic over the appliances. Later communications will be provided when this occurs. **T**o minimize disruptions, we’ve scheduled the partial deployment during non-peak hours. While brief traffic blips are possible, we’re taking all necessary precautions to limit such disruptions. Feb 19, 20:16:33 GMT+0 - Identified - Maintenance is now in progress Feb 19, 20:16:45 GMT+0 - Identified - We are moving some internal traffic within the next 10 minutes. Please note that there is a possibility for disruptions to occur. Type: Maintenance

Duration: 1 hour and 46 minutes

Affected Components: , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>22:02:39</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>20:14:33</var> GMT+0</small><br><strong>Identified</strong> - After the successful testing of our on-site DDoS protection into our network core in the last maintenance, we are moving onto stage 3, which is the beginning of moving production traffic to the protection appliances. We are keeping production traffic to a minimum to ensure no adverse effects in the initial rollout. Over the next few days, we will be shifting more traffic over the appliances. Later communications will be provided when this occurs. **T**o minimize disruptions, we’ve scheduled the partial deployment during non-peak hours. While brief traffic blips are possible, we’re taking all necessary precautions to limit such disruptions..</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>20:16:33</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>20:16:45</var> GMT+0</small><br><strong>Identified</strong> - We are moving some internal traffic within the next 10 minutes. Please note that there is a possibility for disruptions to occur..</p> ]]> Wed, 19 Feb 2025 20:14:33 +0000 https://scala.instatus.com/maintenance/cm7ccsn90000yjw2c3juv60oz https://scala.instatus.com/maintenance/cm7ccsn90000yjw2c3juv60oz Path-Routed Services - Chicago PoP Migration Type: Maintenance Affected Components: Website, Jetbrains Hub Service, Jetbrains TeamCity Service, CHI-D-01, Jetbrains YouTrack Service, jFrog Artifactory Service, Grafana, Sentry Services, CH-VM-01, Pterodactyl Panel, CH-VM-02 Feb 18, 20:27:00 GMT+0 - Identified - One of our upstream providers, Path, is actively restoring their **Chicago PoP**, and we expect minimal disruptions throughout the process. The only potential brief interruption will occur during the final activation phase when traffic is routed back through Chicago. We are expecting updates as work continues, this will be the last PoP that needs to be migrated in the US until Phoenix, however Phoenix is TBD and is not expected to be migrated for a while. Once live, **we will transition away from Kansas City (KC)**, and latency should begin to normalize. Type: Maintenance

Affected Components: , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 18</var>, <var data-var='time'>20:27:00</var> GMT+0</small><br><strong>Identified</strong> - One of our upstream providers, Path, is actively restoring their **Chicago PoP**, and we expect minimal disruptions throughout the process. The only potential brief interruption will occur during the final activation phase when traffic is routed back through Chicago. We are expecting updates as work continues, this will be the last PoP that needs to be migrated in the US until Phoenix, however Phoenix is TBD and is not expected to be migrated for a while. Once live, **we will transition away from Kansas City (KC)**, and latency should begin to normalize..</p> ]]> Tue, 18 Feb 2025 20:27:00 +0000 https://scala.instatus.com/maintenance/cm7cd8h3c0015rqu08u1ngc2h https://scala.instatus.com/maintenance/cm7cd8h3c0015rqu08u1ngc2h Path-Routed Services - Infrequent TCP Issues Type: Incident Affected Components: Website, Jetbrains Hub Service, Jetbrains TeamCity Service, CHI-D-01, Jetbrains YouTrack Service, jFrog Artifactory Service, Grafana, Sentry Services, CH-VM-01, Pterodactyl Panel, CH-VM-02 Feb 1, 21:13:00 GMT+0 - Resolved - Our monitoring systems have detected TCP issues on Path-routed services. We have been in contact with them regarding the ongoing issues. Our team is continuing to monitor and will provide updates as needed. We apologize for the inconvenience. Feb 13, 18:57:24 GMT+0 - Resolved - Path-routed services have completed their maintenance. Our team is actively monitoring for any latency issues and ensuring system stability. We remain hopeful for smooth performance and will provide further updates as needed. Thank you for your patience. Type: Incident

Affected Components: , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>21:13:00</var> GMT+0</small><br><strong>Resolved</strong> - Our monitoring systems have detected TCP issues on Path-routed services. We have been in contact with them regarding the ongoing issues. Our team is continuing to monitor and will provide updates as needed. We apologize for the inconvenience..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>18:57:24</var> GMT+0</small><br><strong>Resolved</strong> - Path-routed services have completed their maintenance. Our team is actively monitoring for any latency issues and ensuring system stability. We remain hopeful for smooth performance and will provide further updates as needed. Thank you for your patience..</p> ]]> Sat, 1 Feb 2025 21:13:00 +0000 https://scala.instatus.com/incident/cm6zjw720004c11ygaa2zag08 https://scala.instatus.com/incident/cm6zjw720004c11ygaa2zag08