Typesense Status - Incident history https://cloud-status.typesense.org Typesense Sun, 31 May 2026 02:00:00 +0000 Issues with new cluster provisioning Type: Incident Duration: 23 hours and 30 minutes Affected Components: Management Console May 31, 13:30:00 GMT+0 - Resolved - The issue is now resolved and New Cluster Provisioning is operational once again. May 31, 02:00:00 GMT+0 - Investigating - We've identified an issue with provisioning new clusters. Existing clusters are unaffected, and configuration changes on existing clusters continue to work normally. Type: Incident

Duration: 23 hours and 30 minutes

Affected Components:

May 31, 13:30:00 GMT+0
Resolved - The issue is now resolved and New Cluster Provisioning is operational once again..

May 31, 02:00:00 GMT+0
Investigating - We've identified an issue with provisioning new clusters. Existing clusters are unaffected, and configuration changes on existing clusters continue to work normally..

]]> Sun, 31 May 2026 02:00:00 +0000 https://cloud-status.typesense.org/incident/cmptud5g90kdantca8806rydi https://cloud-status.typesense.org/incident/cmptud5g90kdantca8806rydi Connectivity issues in two data centers in the Bahrain region Type: Incident Duration: 3 days, 20 hours and 54 minutes Aug 4, 01:59:26 GMT+0 - Resolved - Our upstream infrastructure provider has indicated that the UAE and Bahrain regions are not expected to come back online in the near future. So we've decided to retire these regions from Typesense Cloud. Please see past updates on guidance on what to do. Apr 1, 21:02:03 GMT+0 - Identified - A 2nd data center in the Bahrain region has been affected due to the ongoing conflict. We recommend customers provision a new cluster in a different region, and re-index their data from their primary database, and then update their applications to use the new cluster. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of the Bahrain and UAE regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster. Mar 9, 19:21:27 GMT+0 - Identified - The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster. Mar 3, 03:34:57 GMT+0 - Identified - Our infrastructure provider has completed their initial assessment, and confirmed that one of the three data centers in the Bahrain region have sustained structural damage due to drone strikes in the area. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. HA clusters in this region continue to operate fine with two out of three nodes still functioning. However, given the volatility of the conflicts in the area, we recommend that users with clusters in this region plan to provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Mar 2, 05:05:00 GMT+0 - Investigating - We are observing upstream network connectivity issues in one of the three data centers in the Bahrain region. Typesense Cloud clusters with High Availability enabled remain unaffected, as traffic is being routed away from the affected data center. Typesense Cloud clusters that do not have High Availability enabled will remain unavailable until network connectivity is restored. Type: Incident

Duration: 3 days, 20 hours and 54 minutes

Aug 4, 01:59:26 GMT+0
Resolved - Our upstream infrastructure provider has indicated that the UAE and Bahrain regions are not expected to come back online in the near future. So we've decided to retire these regions from Typesense Cloud. Please see past updates on guidance on what to do. .

Apr 1, 21:02:03 GMT+0
Identified - A 2nd data center in the Bahrain region has been affected due to the ongoing conflict. We recommend customers provision a new cluster in a different region, and re-index their data from their primary database, and then update their applications to use the new cluster. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of the Bahrain and UAE regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster..

Mar 9, 19:21:27 GMT+0
Identified - The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster. .

Mar 3, 03:34:57 GMT+0
Identified - Our infrastructure provider has completed their initial assessment, and confirmed that one of the three data centers in the Bahrain region have sustained structural damage due to drone strikes in the area. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. HA clusters in this region continue to operate fine with two out of three nodes still functioning. However, given the volatility of the conflicts in the area, we recommend that users with clusters in this region plan to provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. .

Mar 2, 05:05:00 GMT+0
Investigating - We are observing upstream network connectivity issues in one of the three data centers in the Bahrain region. Typesense Cloud clusters with High Availability enabled remain unaffected, as traffic is being routed away from the affected data center. Typesense Cloud clusters that do not have High Availability enabled will remain unavailable until network connectivity is restored..

]]> Mon, 2 Mar 2026 05:05:00 +0000 https://cloud-status.typesense.org/incident/cmm8qtg6j04ca9yh84t672vb0 https://cloud-status.typesense.org/incident/cmm8qtg6j04ca9yh84t672vb0 Power Outage in two data centers in the UAE Type: Incident Duration: 4 days, 13 hours and 28 minutes Aug 4, 01:59:10 GMT+0 - Resolved - Our upstream infrastructure provider has indicated that the UAE and Bahrain regions are not expected to come back online in the near future. So we've decided to retire these regions from Typesense Cloud. Please see past updates on guidance on what to do. Mar 9, 19:22:10 GMT+0 - Identified - The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster. Mar 3, 03:31:14 GMT+0 - Identified - Our infrastructure provider has completed their initial assessment, and confirmed that the two data centers in the UAE region have sustained structural damage due to drone strikes. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. Like mentioned earlier, we recommend that users with clusters in this region provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Mar 1, 12:30:58 GMT+0 - Identified - A second data center in the UAE region has lost power, due to active military conflict in the region. An HA Typesense Cluster with 3 nodes can survive a maximum of 1 node failure, and with 2 data centers in the region now down, 2 nodes in HA clusters in the region are now inaccessible, which makes the 3rd node stop serving traffic to avoid split brain scenarios. Our infrastructure provider estimates that it might take at least a day to evaluate critical systems and their safety, before providing an ETA for full restoration. Given the severity and uncertainty of this event, we strongly recommend: 1. Provisioning a new cluster in a different region outside the Middle East. 2. Re-indexing data from your primary database. 3. Updating your application configuration to point to the new cluster. Please note: the cluster cloning feature is unavailable because the regional control plane is also affected. Mar 1, 12:30:57 GMT+0 - Identified - Our infrastructure provider is experiencing a power outage in one of the three data centers in the UAE region. Customers who have enabled High Availability for their Typesense Cloud Cluster continue to operate normally even with one node in the affected data center being down, as traffic is being re-routed away from the affected data center to the other two nodes in the other data centers in the region. Customers who do not have High Availability enabled for their Typesense Cloud Cluster and have their node running in the affected data center are affected by this issue, and will remain unavailable until power is restored. Type: Incident

Duration: 4 days, 13 hours and 28 minutes

Aug 4, 01:59:10 GMT+0
Resolved - Our upstream infrastructure provider has indicated that the UAE and Bahrain regions are not expected to come back online in the near future. So we've decided to retire these regions from Typesense Cloud. Please see past updates on guidance on what to do. .

Mar 9, 19:22:10 GMT+0
Identified - The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation. So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions. If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster. .

Mar 3, 03:31:14 GMT+0
Identified - Our infrastructure provider has completed their initial assessment, and confirmed that the two data centers in the UAE region have sustained structural damage due to drone strikes. They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation. Like mentioned earlier, we recommend that users with clusters in this region provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions. .

Mar 1, 12:30:58 GMT+0
Identified - A second data center in the UAE region has lost power, due to active military conflict in the region. An HA Typesense Cluster with 3 nodes can survive a maximum of 1 node failure, and with 2 data centers in the region now down, 2 nodes in HA clusters in the region are now inaccessible, which makes the 3rd node stop serving traffic to avoid split brain scenarios. Our infrastructure provider estimates that it might take at least a day to evaluate critical systems and their safety, before providing an ETA for full restoration. Given the severity and uncertainty of this event, we strongly recommend: 1. Provisioning a new cluster in a different region outside the Middle East. 2. Re-indexing data from your primary database. 3. Updating your application configuration to point to the new cluster. Please note: the cluster cloning feature is unavailable because the regional control plane is also affected..

Mar 1, 12:30:57 GMT+0
Identified - Our infrastructure provider is experiencing a power outage in one of the three data centers in the UAE region. Customers who have enabled High Availability for their Typesense Cloud Cluster continue to operate normally even with one node in the affected data center being down, as traffic is being re-routed away from the affected data center to the other two nodes in the other data centers in the region. Customers who do not have High Availability enabled for their Typesense Cloud Cluster and have their node running in the affected data center are affected by this issue, and will remain unavailable until power is restored..

]]> Sun, 1 Mar 2026 12:30:57 +0000 https://cloud-status.typesense.org/incident/cmm7thfgn0oc9tu863kdl6ab7 https://cloud-status.typesense.org/incident/cmm7thfgn0oc9tu863kdl6ab7 Issue with provisioning and terminating clusters in N Virginia Type: Incident Duration: 16 hours and 38 minutes Affected Components: Management Console, N Virginia Oct 21, 00:07:31 GMT+0 - Resolved - The upstream infrastructure has been resolved. We've also reprocessed the backlog of clusters that were provisioned, terminated or reconfigured during this time. Oct 20, 07:30:00 GMT+0 - Identified - Our infrastructure provider is experiencing an issue with their control plane which is preventing us from making changes to infra capacity in the N Virginia region. We’re monitoring the issue closely. Note that searches and writes on existing clusters are unaffected by this issue across all regions. Type: Incident

Duration: 16 hours and 38 minutes

Affected Components: ,

Oct 21, 00:07:31 GMT+0
Resolved - The upstream infrastructure has been resolved. We've also reprocessed the backlog of clusters that were provisioned, terminated or reconfigured during this time. .

Oct 20, 07:30:00 GMT+0
Identified - Our infrastructure provider is experiencing an issue with their control plane which is preventing us from making changes to infra capacity in the N Virginia region. We’re monitoring the issue closely. Note that searches and writes on existing clusters are unaffected by this issue across all regions. .

]]> Mon, 20 Oct 2025 07:30:00 +0000 https://cloud-status.typesense.org/incident/cmgz2xmy900hj135qa35pet9r https://cloud-status.typesense.org/incident/cmgz2xmy900hj135qa35pet9r SSL certificate issue - unable to get local issuer certificate Type: Incident Duration: 16 hours Affected Components: Osaka, Oregon, Sao Paulo, Canada, Tokyo, London, Ohio, N California, Seoul, Frankfurt, Cape Town, Ireland, N Virginia, Sydney, , Stockholm, Singapore, Paris, Milan, Mumbai, Cluster Regions → Jun 3, 19:34:53 GMT+0 - Resolved - We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected. Jun 3, 18:45:00 GMT+0 - Monitoring - As the volume of reports started increasing, to prevent more users on older OS / runtimes from running into this issue, we decided to roll back to the previous SSL certificate temporarily. We've also canceled the scheduled roll out of this new SSL certificate to existing clusters. If you're running into SSL cert issues, and if you provisioned a cluster or made config changes during the time frame of this incident, you want to do a \[rolling restart\](https://cloud-help-center.typesense.org/article/33-restart-typesense-cloud-cluster) to refresh your SSL certificate. Alternatively, the certs will also be automatically rolled back in the next few hours. We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected. Jun 3, 10:20:00 GMT+0 - Identified - We started getting sporadic reports of a few users running into SSL certificate issues when connecting to their Typesense Cloud clusters. On closer investigation, we found that they were trying to connect to Typesense from older operating systems or older runtimes that bundle their own root CAs (like Node.js), which hadn't updated their Root CA store recently. Updating their OS / runtime CA certificates, to the latest version fixed the issue for them. Our SSL provider started using a new Root CA certificate to sign our certificate this year, and this new root CA certificate wasn't available on older runtimes. So manually updating the root CA store on users' computers fixed the issue, so that the SSL cert presented by Typesense Cloud was able to be validated successfully. Jun 3, 03:35:00 GMT+0 - Investigating - We rotate our SSL certificates for Typesense Cloud annually as part of our standard operating procedure. We tested and deployed a new SSL cert for _new_ nodes on Typesense Cloud (provisioned either during new cluster creation or via config changes) at 3:30AM UTC on Jun 2, 2025. We also scheduled the new cert to be pushed live to existing nodes at 12:00AM UTC on Jun 3, 2025. Type: Incident

Duration: 16 hours

Affected Components: , , , , , , , , , , , , , , , , , , ,

Jun 3, 19:34:53 GMT+0
Resolved - We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected..

Jun 3, 18:45:00 GMT+0
Monitoring - As the volume of reports started increasing, to prevent more users on older OS / runtimes from running into this issue, we decided to roll back to the previous SSL certificate temporarily. We've also canceled the scheduled roll out of this new SSL certificate to existing clusters. If you're running into SSL cert issues, and if you provisioned a cluster or made config changes during the time frame of this incident, you want to do a \[rolling restart\](https://cloud-help-center.typesense.org/article/33-restart-typesense-cloud-cluster) to refresh your SSL certificate. Alternatively, the certs will also be automatically rolled back in the next few hours. We're now going to be working with our SSL cert provider to identify how to best roll out this new Root CA without affecting older runtimes. This issue only affected: 1) Clusters that were provisioned during the duration of this incident 2) Clusters whose configuration (RAM, CPU, Typesense Version, etc) was changed during the duration of this incident. Existing clusters that were already running prior to the duration of this incident were unaffected..

Jun 3, 10:20:00 GMT+0
Identified - We started getting sporadic reports of a few users running into SSL certificate issues when connecting to their Typesense Cloud clusters. On closer investigation, we found that they were trying to connect to Typesense from older operating systems or older runtimes that bundle their own root CAs (like Node.js), which hadn't updated their Root CA store recently. Updating their OS / runtime CA certificates, to the latest version fixed the issue for them. Our SSL provider started using a new Root CA certificate to sign our certificate this year, and this new root CA certificate wasn't available on older runtimes. So manually updating the root CA store on users' computers fixed the issue, so that the SSL cert presented by Typesense Cloud was able to be validated successfully..

Jun 3, 03:35:00 GMT+0
Investigating - We rotate our SSL certificates for Typesense Cloud annually as part of our standard operating procedure. We tested and deployed a new SSL cert for _new_ nodes on Typesense Cloud (provisioned either during new cluster creation or via config changes) at 3:30AM UTC on Jun 2, 2025. We also scheduled the new cert to be pushed live to existing nodes at 12:00AM UTC on Jun 3, 2025..

]]> Tue, 3 Jun 2025 03:35:00 +0000 https://cloud-status.typesense.org/incident/cmbgwo2e2003aosd29gxm6920 https://cloud-status.typesense.org/incident/cmbgwo2e2003aosd29gxm6920 Cluster resource usage metrics are unavailable in the Web Console UI Type: Incident Duration: 11 hours and 44 minutes Affected Components: Management Console Mar 2, 04:12:30 GMT+0 - Resolved - The issue is now resolved. Metrics are now visible again in the Typesense Cloud Web console. Mar 1, 16:28:42 GMT+0 - Identified - The metrics section in the Typesense Cloud Web console UI is currently unavailable, due to issues in the underlying metrics datastore. We're working on addressing the issue. This issue does not affect any cluster APIs (searches, writes, etc) as customer clusters run on completely independent infrastructure. This issue is isolated to the metrics datastore in the control plane. Type: Incident

Duration: 11 hours and 44 minutes

Affected Components:

Mar 2, 04:12:30 GMT+0
Resolved - The issue is now resolved. Metrics are now visible again in the Typesense Cloud Web console..

Mar 1, 16:28:42 GMT+0
Identified - The metrics section in the Typesense Cloud Web console UI is currently unavailable, due to issues in the underlying metrics datastore. We're working on addressing the issue. This issue does not affect any cluster APIs (searches, writes, etc) as customer clusters run on completely independent infrastructure. This issue is isolated to the metrics datastore in the control plane..

]]> Sat, 1 Mar 2025 16:28:42 +0000 https://cloud-status.typesense.org/incident/cm7qf2lmd00fjahhr1vk21m9v https://cloud-status.typesense.org/incident/cm7qf2lmd00fjahhr1vk21m9v Provisioning issue in Stockholm region Type: Incident Duration: 1 hour and 44 minutes Affected Components: Stockholm Feb 14, 00:58:41 GMT+0 - Resolved - This incident has been resolved. Feb 13, 23:15:00 GMT+0 - Identified - We're currently running into issues provisioning new nodes in the Stockholm region. Existing clusters are unaffected. This only affects new cluster provisioning and config changes. Type: Incident

Duration: 1 hour and 44 minutes

Affected Components:

Feb 14, 00:58:41 GMT+0
Resolved - This incident has been resolved..

Feb 13, 23:15:00 GMT+0
Identified - We're currently running into issues provisioning new nodes in the Stockholm region. Existing clusters are unaffected. This only affects new cluster provisioning and config changes. .

]]> Thu, 13 Feb 2025 23:15:00 +0000 https://cloud-status.typesense.org/incident/cm740es9d0049xhaaaj00tiut https://cloud-status.typesense.org/incident/cm740es9d0049xhaaaj00tiut Typesense Cloud web console is down Type: Incident Duration: 1 hour and 39 minutes Affected Components: Management Console Nov 14, 00:07:06 GMT+0 - Resolved - The issue is fully resolved now, and the web console is functioning normally again. Customers' search clusters were unaffected throughout the duration of this event, as they run on completely independent infrastructure. Nov 13, 22:48:26 GMT+0 - Monitoring - We've identified the issue and have put mitigations in place. So the web console should be functional now as of 5 minutes ago, and we're continuing to monitor how the mitigation performs. Nov 13, 22:27:51 GMT+0 - Investigating - We are currently investigating this incident. Customer's clusters are not affected by this issue, as they run on a different infrastructure and continue to service requests without any interruption. This issue only affects the infrastructure that the web console is running on. Type: Incident

Duration: 1 hour and 39 minutes

Affected Components:

Nov 14, 00:07:06 GMT+0
Resolved - The issue is fully resolved now, and the web console is functioning normally again. Customers' search clusters were unaffected throughout the duration of this event, as they run on completely independent infrastructure..

Nov 13, 22:48:26 GMT+0
Monitoring - We've identified the issue and have put mitigations in place. So the web console should be functional now as of 5 minutes ago, and we're continuing to monitor how the mitigation performs. .

Nov 13, 22:27:51 GMT+0
Investigating - We are currently investigating this incident. Customer's clusters are not affected by this issue, as they run on a different infrastructure and continue to service requests without any interruption. This issue only affects the infrastructure that the web console is running on..

]]> Wed, 13 Nov 2024 22:27:51 +0000 https://cloud-status.typesense.org/incident/cm3ggcgt7001i14d75mzx3jzy https://cloud-status.typesense.org/incident/cm3ggcgt7001i14d75mzx3jzy Cluster Provisioning and Configuration Change Issues Type: Incident Duration: 1 hour and 55 minutes Affected Components: Ireland, Osaka, Oregon, Sao Paulo, Canada, Mumbai, Tokyo, London, N Virginia, Ohio, Stockholm, Milan, Singapore, Sydney, N California, Frankfurt, Seoul, Paris, Cape Town Aug 29, 11:29:23 GMT+0 - Resolved - The issue is now resolved. The upstream APIs are now back up and we've verified that new cluster provisioning and configuration changes now work. Aug 29, 10:08:04 GMT+0 - Identified - We've tracked down the issue and have identified the root cause as originating from our upstream infrastructure provider's APIs being down. This is causing any provisioning of new nodes, both during new cluster provisioning and during configuration changes, to be affected. Existing clusters that are already provisioned and running are unaffected. We are closely monitoring the upstream issue and will post updates as a fix is being worked on. Aug 29, 09:34:00 GMT+0 - Investigating - We're investigating reports of issues with cluster provisioning in Typesense Cloud. Type: Incident

Duration: 1 hour and 55 minutes

Affected Components: , , , , , , , , , , , , , , , , , ,

Aug 29, 11:29:23 GMT+0
Resolved - The issue is now resolved. The upstream APIs are now back up and we've verified that new cluster provisioning and configuration changes now work..

Aug 29, 10:08:04 GMT+0
Identified - We've tracked down the issue and have identified the root cause as originating from our upstream infrastructure provider's APIs being down. This is causing any provisioning of new nodes, both during new cluster provisioning and during configuration changes, to be affected. Existing clusters that are already provisioned and running are unaffected. We are closely monitoring the upstream issue and will post updates as a fix is being worked on..

Aug 29, 09:34:00 GMT+0
Investigating - We're investigating reports of issues with cluster provisioning in Typesense Cloud..

]]> Thu, 29 Aug 2024 09:34:00 +0000 https://cloud-status.typesense.org/incident/cm0f38k9h0002l1q8uvads739 https://cloud-status.typesense.org/incident/cm0f38k9h0002l1q8uvads739 Cluster provisioning issues in Northern Virginia Type: Incident Duration: 2 hours and 15 minutes Affected Components: N Virginia Mar 26, 22:30:00 GMT+0 - Resolved - This incident has been resolved. Mar 26, 21:44:02 GMT+0 - Identified - We've identified the issue and are working towards a resolution. Mar 26, 20:15:00 GMT+0 - Investigating - Provisioning new clusters and node upgrades in the N Virginia region are currently experiencing issues. We are investigating the errors currently. This issue does not affect existing running clusters. Type: Incident

Duration: 2 hours and 15 minutes

Affected Components:

Mar 26, 22:30:00 GMT+0
Resolved - This incident has been resolved..

Mar 26, 21:44:02 GMT+0
Identified - We've identified the issue and are working towards a resolution..

Mar 26, 20:15:00 GMT+0
Investigating - Provisioning new clusters and node upgrades in the N Virginia region are currently experiencing issues. We are investigating the errors currently. This issue does not affect existing running clusters. .

]]> Tue, 26 Mar 2024 20:15:00 +0000 https://cloud-status.typesense.org/incident/clu8ubg81100860b1myh2rsk9ti https://cloud-status.typesense.org/incident/clu8ubg81100860b1myh2rsk9ti Infrastructure issues in one data center in London Type: Incident Duration: 45 minutes Affected Components: London Mar 16, 20:05:00 GMT+0 - Resolved - This incident has been resolved. Mar 16, 19:20:00 GMT+0 - Identified - Our infrastructure provider in London is experiencing an outage in one data center. This incident only affects Typesense Cloud clusters that do not have High Availability enabled, and are running a single node in this particular data center in London. Typesense Cloud clusters that have High Availability enabled are unaffected, as the other nodes running in unaffected data centers continue serving traffic as normal. We are continuing to monitor the incident. Type: Incident

Duration: 45 minutes

Affected Components:

Mar 16, 20:05:00 GMT+0
Resolved - This incident has been resolved..

Mar 16, 19:20:00 GMT+0
Identified - Our infrastructure provider in London is experiencing an outage in one data center. This incident only affects Typesense Cloud clusters that do not have High Availability enabled, and are running a single node in this particular data center in London. Typesense Cloud clusters that have High Availability enabled are unaffected, as the other nodes running in unaffected data centers continue serving traffic as normal. We are continuing to monitor the incident. .

]]> Sat, 16 Mar 2024 19:20:00 +0000 https://cloud-status.typesense.org/incident/cltuim3ia176446bfn5r5yd2zlj https://cloud-status.typesense.org/incident/cltuim3ia176446bfn5r5yd2zlj Cloud Web Dashboard is sporadically unavailable Type: Incident Duration: 3 hours and 21 minutes Affected Components: Management Console Feb 26, 20:16:18 GMT+0 - Resolved - This incident has been resolved. Feb 26, 19:03:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Feb 26, 16:55:00 GMT+0 - Identified - We're having brief blips in the Cloud Dashboard Web UI, returning HTTP 500s. We've identified the root cause of the issue, and have temporary mitigations in place, while we work on the fix. The issue only affects the web admin console, and does NOT affect any running clusters. So search queries, writes etc to the cluster hostnames are working as usual without any issues. Type: Incident

Duration: 3 hours and 21 minutes

Affected Components:

Feb 26, 20:16:18 GMT+0
Resolved - This incident has been resolved..

Feb 26, 19:03:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Feb 26, 16:55:00 GMT+0
Identified - We're having brief blips in the Cloud Dashboard Web UI, returning HTTP 500s. We've identified the root cause of the issue, and have temporary mitigations in place, while we work on the fix. The issue only affects the web admin console, and does NOT affect any running clusters. So search queries, writes etc to the cluster hostnames are working as usual without any issues..

]]> Mon, 26 Feb 2024 16:55:00 +0000 https://cloud-status.typesense.org/incident/clt36wccx101118bbolw258g0ne https://cloud-status.typesense.org/incident/clt36wccx101118bbolw258g0ne Cloud Dashboard in unavailable. Type: Incident Duration: 8 minutes Affected Components: Management Console Sep 15, 12:18:21 GMT+0 - Resolved - This incident has been resolved. Sep 15, 12:09:51 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 8 minutes

Affected Components:

Sep 15, 12:18:21 GMT+0
Resolved - This incident has been resolved..

Sep 15, 12:09:51 GMT+0
Investigating - We are currently investigating this incident..

]]> Fri, 15 Sep 2023 12:09:51 +0000 https://cloud-status.typesense.org/incident/clmkk4p9x11779bqnc3judp5do https://cloud-status.typesense.org/incident/clmkk4p9x11779bqnc3judp5do HTTP 503 on Typesense Cloud web console Type: Incident Affected Components: Management Console Jul 27, 17:05:00 GMT+0 - Resolved - An upgrade to our DB capacity caused unexpected 503s to show up on the web console for about 5 minutes. The issue only affected the web UI and did not affect customer Typesense Cloud clusters. Type: Incident

Affected Components:

Jul 27, 17:05:00 GMT+0
Resolved - An upgrade to our DB capacity caused unexpected 503s to show up on the web console for about 5 minutes. The issue only affected the web UI and did not affect customer Typesense Cloud clusters. .

]]> Thu, 27 Jul 2023 17:05:00 +0000 https://cloud-status.typesense.org/incident/clklexlcw83712bmopr5asp5ad https://cloud-status.typesense.org/incident/clklexlcw83712bmopr5asp5ad Infrastructure issue in N Virginia region Type: Incident Affected Components: N Virginia Jun 8, 20:41:16 GMT+0 - Resolved - The issue has been resolved. Type: Incident

Affected Components:

Jun 8, 20:41:16 GMT+0
Resolved - The issue has been resolved. .

]]> Thu, 8 Jun 2023 20:41:16 +0000 https://cloud-status.typesense.org/incident/clinlt1k6260415bxn0uypeuo8f https://cloud-status.typesense.org/incident/clinlt1k6260415bxn0uypeuo8f Infrastructure issue in London region Type: Incident Duration: 34 minutes Affected Components: London May 24, 21:29:22 GMT+0 - Resolved - This incident has been resolved. May 24, 21:09:59 GMT+0 - Monitoring - All affected nodes have recovered and we're continuing to monitor the situation. May 24, 20:30:00 GMT+0 - Identified - One of the data centers in our London region has underlying infrastructure issues. Clusters that have High Availability turned ON are unaffected by this issue as the other nodes in the cluster in other data centers are continuing to service traffic. But clusters with High Availability turned OFF, that are running a single node in this particular data-center in London are currently inaccessible, while the infrastructure recovers. Type: Incident

Duration: 34 minutes

Affected Components:

May 24, 21:29:22 GMT+0
Resolved - This incident has been resolved..

May 24, 21:09:59 GMT+0
Monitoring - All affected nodes have recovered and we're continuing to monitor the situation. .

May 24, 20:30:00 GMT+0
Identified - One of the data centers in our London region has underlying infrastructure issues. Clusters that have High Availability turned ON are unaffected by this issue as the other nodes in the cluster in other data centers are continuing to service traffic. But clusters with High Availability turned OFF, that are running a single node in this particular data-center in London are currently inaccessible, while the infrastructure recovers. .

]]> Wed, 24 May 2023 20:30:00 +0000 https://cloud-status.typesense.org/incident/cli26p3jz1040938ban3a714nvj5 https://cloud-status.typesense.org/incident/cli26p3jz1040938ban3a714nvj5 Typesense Cloud Web Console - Database Upgrade Type: Maintenance Duration: 1 hour and 1 minute Affected Components: Management Console Jan 21, 00:39:51 GMT+0 - Completed - Maintenance has completed successfully. Jan 20, 23:47:49 GMT+0 - Identified - Maintenance is now in progress. Jan 20, 23:38:40 GMT+0 - Identified - We are upgrading our database engine version and capacity. During this time the Typesense Web console will be unavailable. Typesense Cloud clusters will remain unaffected. Type: Maintenance

Duration: 1 hour and 1 minute

Affected Components:

Jan 21, 00:39:51 GMT+0
Completed - Maintenance has completed successfully..

Jan 20, 23:47:49 GMT+0
Identified - Maintenance is now in progress..

Jan 20, 23:38:40 GMT+0
Identified - We are upgrading our database engine version and capacity. During this time the Typesense Web console will be unavailable. Typesense Cloud clusters will remain unaffected..

]]> Fri, 20 Jan 2023 23:38:40 +0000 https://cloud-status.typesense.org/maintenance/cld5613o62172i9nbsm565c44 https://cloud-status.typesense.org/maintenance/cld5613o62172i9nbsm565c44 Typesense Cloud Management Console - HTTP 503 Errors Type: Incident Duration: 4 hours and 18 minutes Affected Components: Management Console Sep 13, 23:33:33 GMT+0 - Resolved - The issue has been resolved fully. Sep 13, 20:06:07 GMT+0 - Monitoring - The downtime was caused due to a recent change in our deployment pipeline. We've added a fix for it and the console is back up now. Sep 13, 19:15:08 GMT+0 - Investigating - We are currently investigating this incident. This issue only affects the Typesense Cloud Management console. Customer Clusters are unaffected by this. Type: Incident

Duration: 4 hours and 18 minutes

Affected Components:

Sep 13, 23:33:33 GMT+0
Resolved - The issue has been resolved fully..

Sep 13, 20:06:07 GMT+0
Monitoring - The downtime was caused due to a recent change in our deployment pipeline. We've added a fix for it and the console is back up now. .

Sep 13, 19:15:08 GMT+0
Investigating - We are currently investigating this incident. This issue only affects the Typesense Cloud Management console. Customer Clusters are unaffected by this. .

]]> Tue, 13 Sep 2022 19:15:08 +0000 https://cloud-status.typesense.org/incident/cl80kpymk0852hkofpb2u6kq9 https://cloud-status.typesense.org/incident/cl80kpymk0852hkofpb2u6kq9 Outage in one of the three data center regions in London region Type: Incident Duration: 3 hours and 9 minutes Affected Components: London Jul 10, 21:08:47 GMT+0 - Resolved - The issue is now resolved Jul 10, 18:00:00 GMT+0 - Identified - Our infrastructure provider has run into a data-center-wide thermal issue in one of the three data centers in the London region. They've already identified the issue and are working on a resolution, with an unknown ETA. Highly-Available Typesense Cloud clusters with nodes in multiple data centers / regions continue to operate without any issues. Single node clusters in this particular data center will be unaccessible until the infrastructure issue is fixed. Type: Incident

Duration: 3 hours and 9 minutes

Affected Components:

Jul 10, 21:08:47 GMT+0
Resolved - The issue is now resolved.

Jul 10, 18:00:00 GMT+0
Identified - Our infrastructure provider has run into a data-center-wide thermal issue in one of the three data centers in the London region. They've already identified the issue and are working on a resolution, with an unknown ETA. Highly-Available Typesense Cloud clusters with nodes in multiple data centers / regions continue to operate without any issues. Single node clusters in this particular data center will be unaccessible until the infrastructure issue is fixed..

]]> Sun, 10 Jul 2022 18:00:00 +0000 https://cloud-status.typesense.org/incident/cl5fo58ic17942ffoisy8jpou3 https://cloud-status.typesense.org/incident/cl5fo58ic17942ffoisy8jpou3 New Cluster Provisioning issue in Oregon Region Type: Incident Duration: 20 hours and 29 minutes Affected Components: Management Console, Oregon Feb 9, 08:00:30 GMT+0 - Resolved - The issue has been resolved. Feb 8, 11:32:00 GMT+0 - Identified - We've run into a rate limit with our infra provider in the Oregon region, and this is affecting the ability to provision new clusters in this region (only). Running clusters are unaffected by this issue. We are working with our infrastructure provider to resolve this issue. In the meantime, we've changed the default region for new clusters to Northern California. Type: Incident

Duration: 20 hours and 29 minutes

Affected Components: ,

Feb 9, 08:00:30 GMT+0
Resolved - The issue has been resolved..

Feb 8, 11:32:00 GMT+0
Identified - We've run into a rate limit with our infra provider in the Oregon region, and this is affecting the ability to provision new clusters in this region (only). Running clusters are unaffected by this issue. We are working with our infrastructure provider to resolve this issue. In the meantime, we've changed the default region for new clusters to Northern California. .

]]> Tue, 8 Feb 2022 11:32:00 +0000 https://cloud-status.typesense.org/incident/ckze6mevv1461830ebn5ivtb7ss8 https://cloud-status.typesense.org/incident/ckze6mevv1461830ebn5ivtb7ss8 Infrastructure issues in Northern Virginia region Type: Incident Duration: 11 hours and 44 minutes Affected Components: Management Console, N Virginia Dec 8, 03:44:12 GMT+0 - Resolved - Our infrastructure provider is back up in the Northern Virgina region, we're continuing to monitor for any lingering issues. Dec 7, 21:14:58 GMT+0 - Identified - We are continuing to monitor the situation with our infrastructure provider as they work towards a resolution Dec 7, 16:00:00 GMT+0 - Investigating - Our infrastructure provider in the Northern Virgina region is currently experiencing issues. We are closely monitoring the situation. This currently affects new cluster provisioning and nodes in the N Virginia regions. All other nodes and clusters in other regions are unaffected. Type: Incident

Duration: 11 hours and 44 minutes

Affected Components: ,

Dec 8, 03:44:12 GMT+0
Resolved - Our infrastructure provider is back up in the Northern Virgina region, we're continuing to monitor for any lingering issues. .

Dec 7, 21:14:58 GMT+0
Identified - We are continuing to monitor the situation with our infrastructure provider as they work towards a resolution.

Dec 7, 16:00:00 GMT+0
Investigating - Our infrastructure provider in the Northern Virgina region is currently experiencing issues. We are closely monitoring the situation. This currently affects new cluster provisioning and nodes in the N Virginia regions. All other nodes and clusters in other regions are unaffected..

]]> Tue, 7 Dec 2021 16:00:00 +0000 https://cloud-status.typesense.org/incident/ckwwazu5q3875281occ2nedovc https://cloud-status.typesense.org/incident/ckwwazu5q3875281occ2nedovc Cluster Status in Typesense Cloud dashboard is returning "Unhealthy" even for Healthy clusters Type: Incident Duration: 27 minutes Affected Components: Management Console Dec 3, 01:07:59 GMT+0 - Resolved - We've pushed out a fix for the issue. The Typesense Cloud dashboard should now show the correct cluster health status. This issue was isolated to just the Cloud dashboard display and cluster uptime was not affected. Dec 3, 00:41:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 27 minutes

Affected Components:

Dec 3, 01:07:59 GMT+0
Resolved - We've pushed out a fix for the issue. The Typesense Cloud dashboard should now show the correct cluster health status. This issue was isolated to just the Cloud dashboard display and cluster uptime was not affected..

Dec 3, 00:41:00 GMT+0
Investigating - We are currently investigating this incident..

]]> Fri, 3 Dec 2021 00:41:00 +0000 https://cloud-status.typesense.org/incident/ckwpoitq1503977rn5o6jnaczk https://cloud-status.typesense.org/incident/ckwpoitq1503977rn5o6jnaczk Upgrading Database Capacity Type: Incident Duration: 1 day Affected Components: Management Console Jul 18, 22:04:00 GMT+0 - Resolved - The database maintenance is now complete. Jul 18, 21:42:00 GMT+0 - Identified - We're upgrading our database capacity and you might see blips when viewing the Typesense Cloud Dashboard and website. This does NOT affect Typesense Cloud clusters. Type: Incident

Duration: 1 day

Affected Components:

Jul 18, 22:04:00 GMT+0
Resolved - The database maintenance is now complete..

Jul 18, 21:42:00 GMT+0
Identified - We're upgrading our database capacity and you might see blips when viewing the Typesense Cloud Dashboard and website. This does NOT affect Typesense Cloud clusters..

]]> Sun, 18 Jul 2021 22:04:00 +0000 https://cloud-status.typesense.org/incident/ckrj3ruku1437387tlmixoo840e https://cloud-status.typesense.org/incident/ckrj3ruku1437387tlmixoo840e Intermittent HTTP 503s with Cloud Dashboard Type: Incident Duration: 6 days and 15 hours Affected Components: Management Console Jul 6, 04:09:00 GMT+0 - Resolved - This incident has been resolved. Jul 6, 02:56:00 GMT+0 - Monitoring - A fix has been implemented and we are monitoring the results. Jul 6, 02:00:00 GMT+0 - Identified - We've identified the issue is being caused by an overloaded server. We are working on mitigating the issue. Jul 6, 01:30:00 GMT+0 - Investigating - When visiting cloud.typesense.org, some users are intermittently seeing HTTP 503s when loading the page. Refreshing the page a few times resolves the issue. This only affects the Typesense Cloud management dashboard, and not the actual Typesense Clusters. We are currently investigating this issue. Type: Incident

Duration: 6 days and 15 hours

Affected Components:

Jul 6, 04:09:00 GMT+0
Resolved - This incident has been resolved. .

Jul 6, 02:56:00 GMT+0
Monitoring - A fix has been implemented and we are monitoring the results. .

Jul 6, 02:00:00 GMT+0
Identified - We've identified the issue is being caused by an overloaded server. We are working on mitigating the issue. .

Jul 6, 01:30:00 GMT+0
Investigating - When visiting cloud.typesense.org, some users are intermittently seeing HTTP 503s when loading the page. Refreshing the page a few times resolves the issue. This only affects the Typesense Cloud management dashboard, and not the actual Typesense Clusters. We are currently investigating this issue. .

]]> Tue, 6 Jul 2021 01:30:00 +0000 https://cloud-status.typesense.org/incident/ckrj3uqta1658407tlm1do1d2x8 https://cloud-status.typesense.org/incident/ckrj3uqta1658407tlm1do1d2x8 Infrastructure issues in Frankfurt data center Type: Incident Duration: 6 hours and 3 minutes Affected Components: Frankfurt Jun 11, 02:33:00 GMT+0 - Resolved - This incident has been resolved. Jun 11, 01:22:00 GMT+0 - Monitoring - Our infrastructure provider has largely recovered from the hardware issues. We continue to follow and monitor the incident closely. All affected Typesense clusters on the Frankfurt region have recovered now. Jun 10, 20:30:00 GMT+0 - Identified - Our infrastructure provider is experiencing hardware issues in one of their data centers in Frankfurt. This is impacting some customers who have provisioned a single-node Typesense cluster in that data center. We're watching for updates from our infra provider and will update this incident as we learn more. Type: Incident

Duration: 6 hours and 3 minutes

Affected Components:

Jun 11, 02:33:00 GMT+0
Resolved - This incident has been resolved. .

Jun 11, 01:22:00 GMT+0
Monitoring - Our infrastructure provider has largely recovered from the hardware issues. We continue to follow and monitor the incident closely. All affected Typesense clusters on the Frankfurt region have recovered now. .

Jun 10, 20:30:00 GMT+0
Identified - Our infrastructure provider is experiencing hardware issues in one of their data centers in Frankfurt. This is impacting some customers who have provisioned a single-node Typesense cluster in that data center. We're watching for updates from our infra provider and will update this incident as we learn more. .

]]> Thu, 10 Jun 2021 20:30:00 +0000 https://cloud-status.typesense.org/incident/ckrj44nsc1161677yom803ldqyg https://cloud-status.typesense.org/incident/ckrj44nsc1161677yom803ldqyg Javascript errors on web console Type: Incident Duration: 59 minutes Affected Components: Management Console May 17, 19:59:00 GMT+0 - Resolved - This incident has been resolved. May 17, 19:01:00 GMT+0 - Identified - We are continuing to work on a fix for this issue. May 17, 19:00:00 GMT+0 - Investigating - One of our web console servers ran out of memory during a deployment. We're in the process of upscaling it. This only affects the web console and Typesense Cloud clusters are unaffected. Type: Incident

Duration: 59 minutes

Affected Components:

May 17, 19:59:00 GMT+0
Resolved - This incident has been resolved. .

May 17, 19:01:00 GMT+0
Identified - We are continuing to work on a fix for this issue. .

May 17, 19:00:00 GMT+0
Investigating - One of our web console servers ran out of memory during a deployment. We're in the process of upscaling it. This only affects the web console and Typesense Cloud clusters are unaffected. .

]]> Mon, 17 May 2021 19:00:00 +0000 https://cloud-status.typesense.org/incident/ckrj492xa34285uvncj10agiqc https://cloud-status.typesense.org/incident/ckrj492xa34285uvncj10agiqc