tag:status.solana.com,2005:/historySolana Status - Incident History2024-03-28T15:47:41ZSolanatag:status.solana.com,2005:Incident/199203172024-02-06T15:09:24Z2024-02-06T15:09:24Zmb-020624<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>15:09</var> UTC</small><br><strong>Resolved</strong> - Block production on Solana mainnet beta resumed at 14:57 UTC, following a successful upgrade to v1.17.20 and a restart of the cluster by validator operators. Engineers will continue to monitor performance as network operations are restored.<br /><br /><br />The outage began at approximately 09:53 UTC, lasting 5 hours. Core contributors are working on a root cause report, which will be made available once complete.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>10:22</var> UTC</small><br><strong>Investigating</strong> - Engineers from across the ecosystem are investigating an outage on mainnet-beta</p>tag:status.solana.com,2005:Incident/162745162023-02-26T02:09:04Z2023-02-26T15:48:39ZCluster Instability<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>02:09</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>01:28</var> UTC</small><br><strong>Update</strong> - The Solana community successfully restarted the cluster - engineers will continue to monitor network performance</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>21:45</var> UTC</small><br><strong>Update</strong> - A coordinated restart is underway to address an issue during the upgrade from 1.13 to 1.14 that caused block finalization to slow significantly.<br /><br />Validators: please follow updated instructions below https://docs.google.com/document/d/e/2PACX-1vQVMKVqh1o-qF4Z8ATM-Qxa_FhT96qoCmqGHFroq4e1EbrHkfg-H8Q8Z_FRnX3mYOxRgBbK6yzDols_/pub</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>12:32</var> UTC</small><br><strong>Monitoring</strong> - A coordinated restart is underway to address an issue during the upgrade from 1.13 to 1.14 that caused block finalization to slow significantly.<br /><br />Validators: please follow instructions below https://docs.google.com/document/d/e/2PACX-1vQzf7XpdoXB99lCWu6na9PhLy4SOx-xcjP1LlGiVR0PB6vpyGpol1c8_PufAa8TZZp_R0NlgaqlC7a1/pub</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>Investigating</strong> - Engineers are currently investigating slow root production on mainnet beta.<br /><br />2023-02-25 05:46:16 UTC the Solana network entered degraded mode (vote-only). This means that economic activity on the network stopped while the network attempts to recover. Core contributors of the protocol are working to identify the root cause of this issue.</p>tag:status.solana.com,2005:Incident/157945492023-01-08T07:01:17Z2023-01-08T07:01:17ZPublic Endpoints and Explorer offline<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>07:01</var> UTC</small><br><strong>Resolved</strong> - The Public RPC and Explorer endpoints are online.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>04:53</var> UTC</small><br><strong>Identified</strong> - Mainnet beta Public RPC endpoints and Explorer are currently offline as RPC node software is upgraded, following a bug in a test release 1.14. This bug impacted other nodes which had adopted the test release. If you are operating a node on a test release, please change over to 1.13. <br /><br />Block production has not been impacted and the Solana network remains online.</p>tag:status.solana.com,2005:Incident/113331672022-10-01T07:06:06Z2022-10-01T07:06:06ZDegraded Performance<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>07:06</var> UTC</small><br><strong>Resolved</strong> - Validator operators successfully completed a cluster restart of Mainnet Beta at 7 AM UTC.<br /><br />Network operators an dapps will continue to restore client services over the next several hours.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>00:46</var> UTC</small><br><strong>Update</strong> - The Solana network is experiencing an outage and not processing transactions. Developers across the ecosystem are working on diagnosing the issue and to restart the network. More information will be provided as it becomes available.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Investigating</strong> - The Solana Network is currently experiencing degraded performance. Developers across the ecosystem are working on diagnosing the issue and more information will be provided as it becomes available.</p>tag:status.solana.com,2005:Incident/100769722022-06-06T16:32:07Z2022-06-06T16:32:07ZMainnet Beta Clock Drift<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>03:34</var> UTC</small><br><strong>Monitoring</strong> - On-chain time continues to run behind that of wall clocks, due to longer-than-normal block times. While this has no impact on performance or network operations, the time reported by block explorers and dapps may not reflect wall-clock time.</p>tag:status.solana.com,2005:Incident/101254222022-06-01T21:06:03Z2022-06-01T21:10:39ZMainnet Beta Outage<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>21:06</var> UTC</small><br><strong>Resolved</strong> - Validator operators successfully completed a cluster restart of Mainnet Beta at 9:00 PM UTC, following a roughly 4 and a half hour outage after the network failed to reach consensus. Network operators and dapps will continue to restore client services over the next several hours.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>19:32</var> UTC</small><br><strong>Monitoring</strong> - Mainnet Beta Validators: Please follow the restart instructions linked below, and upgrade 1.9.x to 1.9.28 – If you are on 1.10.x please upgrade to 1.10.23 https://docs.google.com/document/u/1/d/e/2PACX-1vTidsSmhjSxg3r0x0XwicHZfrBCQ_BHVMGl9GKYlvWdpVIXkV-9QXN1hTDyKRwDB7wfoYfwwKs0x9Ix/pub</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>16:55</var> UTC</small><br><strong>Investigating</strong> - Block production on Solana Mainnet Beta has halted. Validator operators should prepare for a restart in mb-validators on Discord. (solana.com/discord)</p>tag:status.solana.com,2005:Incident/101150462022-05-31T18:01:56Z2022-05-31T18:01:56ZDegraded performance<p><small>May <var data-var='date'>31</var>, <var data-var='time'>18:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>13:52</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/101009572022-05-29T21:05:48Z2022-05-29T21:05:48ZDegraded performance<p><small>May <var data-var='date'>29</var>, <var data-var='time'>21:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>18:38</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/100955262022-05-28T18:50:21Z2022-05-28T18:50:21ZDegraded performance<p><small>May <var data-var='date'>28</var>, <var data-var='time'>18:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>15:54</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/100838232022-05-26T23:11:36Z2022-05-26T23:11:36ZDegraded performance<p><small>May <var data-var='date'>26</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:34</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/100688562022-05-25T12:55:59Z2022-05-25T12:55:59ZDegraded performance<p><small>May <var data-var='date'>25</var>, <var data-var='time'>12:55</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>02:16</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/100330542022-05-20T07:47:42Z2022-05-20T07:47:42ZDegraded performance<p><small>May <var data-var='date'>20</var>, <var data-var='time'>07:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>00:48</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/100129212022-05-17T23:38:45Z2022-05-17T23:38:45ZDegraded performance<p><small>May <var data-var='date'>17</var>, <var data-var='time'>23:38</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>16:31</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/99765982022-05-14T03:11:20Z2022-05-14T03:11:20ZDegraded performance<p><small>May <var data-var='date'>14</var>, <var data-var='time'>03:11</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>16:07</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/98877752022-05-01T15:20:59Z2022-05-01T15:20:59ZMainnet Beta Outage<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>15:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>05:31</var> UTC</small><br><strong>Monitoring</strong> - Validator operators successfully completed a cluster restart of Mainnet Beta at 3:00 AM UTC, following a roughly 7 hour outage after the network failed to reach consensus. Network operators an dapps will continue to restore client services over the next several hours.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Identified</strong> - Block production on Solana Mainnet Beta has halted. Validator operators should prepare for a restart in mb-validators on Discord. (solana.com/discord)</p>tag:status.solana.com,2005:Incident/98754562022-04-29T13:56:11Z2022-04-29T13:56:11ZDegraded performance<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>13:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>22:53</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/97592252022-04-13T02:07:19Z2022-04-13T02:07:19ZDegraded performance<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>02:07</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>14:04</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/97429522022-04-11T12:20:39Z2022-04-11T12:20:39ZDegraded performance<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>12:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>12:20</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>00:03</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/97320482022-04-07T22:21:38Z2022-04-07T22:21:38ZDegraded performance<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>22:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/97232162022-04-06T20:57:20Z2022-04-06T20:57:20ZDegraded performance<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>20:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/96813032022-04-01T05:50:08Z2022-04-01T05:50:08ZDegraded performance<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>05:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>01:12</var> UTC</small><br><strong>Investigating</strong> - Solana mainnet-beta is currently experiencing degraded performance.</p>tag:status.solana.com,2005:Incident/96446312022-03-28T12:09:49Z2022-03-28T12:09:49ZSeveral RPC nodes down<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:09</var> UTC</small><br><strong>Resolved</strong> - Most of the affected nodes have been updated to v.1.9 and are running properly now.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>02:35</var> UTC</small><br><strong>Monitoring</strong> - A new feature set was activated during the last epoch boundary and it caused all the v1.8 RPC nodes to fork off. <br />Nodes that had upgraded from v1.9 were not affected</p>tag:status.solana.com,2005:Incident/95964162022-03-22T00:42:02Z2022-03-22T00:42:02ZDegraded performance<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>00:42</var> UTC</small><br><strong>Resolved</strong> - Solana mainnet beta was experiencing high levels of network congestion that caused many transactions to drop at during that time.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>18:29</var> UTC</small><br><strong>Investigating</strong> - The Solana Network is experiencing degraded performance, we are currently investigating.</p>tag:status.solana.com,2005:Incident/95110942022-03-10T19:39:46Z2022-03-10T19:39:46ZDegraded performance<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>19:39</var> UTC</small><br><strong>Resolved</strong> - The Solana Network is stable.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>18:41</var> UTC</small><br><strong>Investigating</strong> - The Solana Network is currently experiencing degraded performance, we are currently investigating.</p>tag:status.solana.com,2005:Incident/95094822022-03-10T16:43:56Z2022-03-10T16:43:56ZDegraded performance<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Resolved</strong> - The Solana Network is stable.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>15:24</var> UTC</small><br><strong>Investigating</strong> - The Solana Network is currently experiencing degraded performance, we are currently investigating.</p>