Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

If you need help subscribing to Outage notifications just email support@helixtelecom.com and we will be happy to setup a subscription for you.

Infrastructure

About

Username
Infrastructure
Joined
Visits
151
Last Active
Roles
Member, Administrator, Moderator

Comments

  • Our staff has received a response and the issue appears to have been resolved.   Any customer still experiencing technical difficulty may need to reboot their device.  If problems persist please visit https://fixmyphonesystem.com and choose one of t…
  • Helix Telecom has isolated the outage to a portion of Century Link's network managed by Level 3 Communications.  Century Link customers, or other carriers who utilize Level 3 services, may be experiencing a loss of connectivity.   Helix Telecom has …
  • Helix Telecom staff completed applying security updates to a number of key systems today.   During this time we found logic problems in the call routing that only occur when both the primary and secondary database clusters are offline.  The problem …
  • The problem was isolated to a Denial of Service event.  Our ISP is investigating the source to help mitigate any potential problems in the future.   Customers who would like more information about what a Denial if Service event is can read this art…
  • Service has been restored but we are not closing this outage to provide our engineer more time to investigate the root cause.  Any customers still experiencing difficulty should disconnect power from their devices, wait a few seconds, reconnect powe…
  • The outage has been identified and resolved.  Any customers still experiencing difficulty should submit a ticket by visiting http://fixmyphonesystem.com.
  • Our primary internet service provider has been able to mitigate the problem but are still investigating the root cause. Customers may still experience connectivity issues but engineers are working on a resolution.
  • The outage has been found to be our primary internet/fiber connection to our facility.  We are working with our upstairs providers to resolve the problem or re-route internet traffic to a secondary ISP.
  • Our engineers made done additional changes and believe the problem has been resolved.  Any customer still experiencing difficulty should send an email to support@helixtelecom.com to make us aware of any dropped calls affecting more than one device s…
  • The change made by our engineers has not resolved the problem.  We are resuming our investigation.
  • Our engineers believe the problem has been resolved.   Any customer still experiencing difficulty should send an email to support@helixtelecom.com to make us aware of any dropped calls affecting more than one device simultaneously.   We are not clos…
  • The cause of this outage has been determined to be a malformed package received by one of our core routers.   The packet contents have been sent upstream to our router vendor and a patch has been applied to resolve the problem.  We are marking the o…
  • Our staff has completed restoration of the Automatic Call Recording server and synchronizing all pending recordings to the new server.   The failure was caused by a corrupted hardware raid controller on the storage server's disk array.  Our monitor…
  • The server responsible for storing all call recordings has failed.  A replacement has been ordered and is scheduled to be installed between May 9th and May 10th.  The process for recovering data from the failed server is ongoing and expected to tak…
  • Our engineers are still working to restore the Automatic Call Recording portal.    Call Recordings continue to be queued.  If an Authorized User on your account needs a specific call recording please submit a ticket including the estimated date, tim…
  • Hello Michaell, Please ask an Authorized User on your account to email this feature request to infrastructure@helixtelecom.com.  Please be sure to include the model of Grandstream phones you are using.  We are happy to update the phone configurati…
  • Helix Telecom staff have completed migration of the Recordings server.   This change has been published to all major ISPs worldwide but may require as long as 48 hours to update.   If you experience difficulty accessing Call Recordings please visit …
  • The call quality issues reported this morning were caused by a malfunctioning router.   Our staff power cycled the router and this resolved the issue.   We are still investigating the root cause of the outage but our staff faced a choice between res…
  • Helix Telecom Monitoring Systems show the issue has been resolved but we are still waiting for confirmation from our staff at the data center before clearing this outage.
  • Service had been restored.  We are investigating the cause of the outage further and we will post more information as soon as it becomes available.  We are sorry for the inconvenience and really appreciate all the kind words from customers who und…
  • We have identified the failed router causing the issue and working to repair the problem.  We should have more information shortly.
  • Our engineers are on-site.  The problem appears to be a failed or hung networking device.  We will post an update when we have additional information.
  • Customers with account numbers beginning with 200 have been restored to the previous DC.   Our staff finished duplicating all the servers to the new location and we will keep these servers ready for the remainder of the day in case the Verizon fiber…
  • Our staff was in the process of moving all of our servers when Verizon notified us the fiber repairs were completed.   They are still testing the repairs and slowly restoring service.  We will need a few more minutes to see if the repairs hold and t…
  • We have moved all customers with account numbers beginning with 200 and we believe those customers are working.   Other customers will follow shortly.   We are anticipating a complete loss of connectivity during the migration. 
  • Verizon is unable to provide us an ETA for a resolution to repair.   For this reason we are in the process of migrating all customers to a new data center.   This process will be invasive and lead to a temporary loss of service.   We will post more …
  • We are aware of reports that our workaround does not address all of our customer's concerns.   The problem has been traced to a Verizon Business fiber connection cut and Verizon has advised us they are working to resolve the problem. 
  • We have identified the cause of the problem and implemented a temporary workaround.  We have also contacted the Internet Service Provider responsible for resolving the issue.   We hope to have more information shortly. 
  • Helix Telecom has eliminated the Dual DC Service on June 15, 2017 due to low enrollment.  Customers currently using the service will remain unaffected until January, 2019.  
  • Service has been restored.   The problem was found to be related to a routing table update.  Backup tables were restored and we are investigating the root cause of the failure.