ALERT: RESOLVED
Outdated Email Alerts Generated on January 20, 2020

01/21 08:15 am -An issue in our communications tool resulted in a number of outdated alerts related to scheduled maintenance and service disruptions to be generated and sent to our customers and partners yesterday. These emails can be ignored; there is no current interruption to our processing or reporting platforms. We are investigating the cause of these resubmissions so we can prevent any further confusion. Our apologies for any inconvenience.

ALERT: COMPLETED
Planned Maintenance Window - January 18, 2020 1:00 - 6:00 am EST

Paya will be performing maintenance to our transaction processing platforms on Saturday, January 18, 2020 between 1:00 am and 6:00 am EST. This scheduled maintenance is part of our ongoing efforts to provide the most robust and stable transaction processing experience possible.

During the maintenance window, transaction processing platforms may be unavailable for up to five (5) hours. Users may be unable to submit transactions, settlements, or reporting inquiries to the Paya Gateway until after maintenance is completed.

Every effort will be made to keep the planned service interruption as brief as possible. We appreciate your understanding and apologize for any inconvenience.

ALERT:
Payment/Vault/Settlement Transaction Latency/Timeout in Paya Gateway
 12/02/2019 - 01/02/2020

01/10 09:00 am EST -We've successfully deployed updates to address latency and timeout issues. Transactions are processing at normal speeds. We will continue to monitor closely.

01/03 09:00 am EST -Processing times are at normal levels this morning. We are continuing to observe and take remedial action when we see times increase.

01/02 01:25 pm EST -Processing times have returned to normal levels. We are continuing to monitor closely for any additional impact.

01/02 12:00 pm EST -Paya is experiencing sporadic spikes in processing times. We are fully engaged and working to remedy the behavior as quickly as possible. Impacted transactions may return a "processing error" or "timeout" or not display any response. Please note that the majority of transactions submitted are, in fact, processing successfully in spite of these types of errors. Please check your Virtual Terminal transaction activity prior to resubmitting any transaction that returns an error or timeout.

12/16 09:10 am EST -Paya has conducted successful testing of a fix expected to bring final resolution. Secondary testing is being conducting prior to deployment. Meanwhile, monitoring and remediation continues. We have seen no timeouts since Friday, December 6, 2019.

12/11 08:00 am EST -Paya is continuing to monitor 24/7. We have not seen any instances of timeouts since the evening of Friday, 12/06, and transaction latency has been reduced significantly, with fewer spikes and shorter delays when latency does exist. Our investigation into the cause of these spikes is ongoing.

12/06 04:40 pm EST -A code deployment made this afternoon did not work as anticipated. We saw a brief spike in processing times following that deployment. The code has been rolled back and latency has returned to normal.

12/06 10:00 am EST -We deployed fixes on the evening of 12/05. By 09:00 pm EST we were no longer experiencing spikes in latency. That has remained true over the last 13 hours. We are continuing to monitor our systems closely to ensure we do not see any new rise processing times.

12/05 03:55 pm EST -Our fixes are still in testing. While that testing continues, we have automated our process of monitoring and correcting latency spikes as they develop. That process was taking approximately 15 minutes per occurrence and is now taking approximately 2 minutes.

12/05 12:00 pm EST -We have identified three sources in our code base that are contributing to the sporadic latency our customers have been experiencing over the last several days. Our IT teams are engaged in building and testing fixes to address those areas and return processing speeds to normal.

 We expect to have  fixes tested today. We will provide an update regarding implementation timing at 3:30 pm EST.

Please contact our Support team at support@paya.com for assistance in reversing any payments that may have been processed more than one time as a result of latency or timeout errors.

12/04 04:30 pm EST -Our investigation to identify the cause of sporadic latency remains ongoing. Our teams are continuing to monitor for occurrences of latency and take corrective action as they happen.

12/04 10:50 am EST -Our investigation to identify the cause of sporadic latency remains ongoing. Our teams are continuing to monitor for occurrences of latency and take corrective action as they happen.

12/03 05:05 pm EST -The source of the latency hasn’t been determined. We’re doing everything we can to find the source and remedy it. We are able to identify the spikes in latency as they are happening and we have teams monitoring our systems 24/7 specifically for these occurrences. Since we cannot predict when a spike will happen, we will always be reactive, and because of that spikes will result in some transactions taking a long time to process or timing out. Timeouts will sometimes lead to customers or consumers retrying transactions, resulting in duplicate payments. Our Support teams can assist in identifying and remedying any duplicate payments.

12/03 02:55 pm EST -We are currently experiencing another spike in transaction processing times that began at approximately 02:28 pm EST.

12/03 01:05 pm EST -We experienced another spike in transaction processing times between 12:22 and 12:52 EST today. Our teams are continuing work to identify the root cause and provide a resolution as quickly as possible.

12/03 12:25 pm EST -Numerous customers have reported transactions submitted more than once as a result of latency issues yesterday and during a brief period this morning. Please check your Virtual Terminal reports prior to re-attempting any transactions that do not appear to complete.

12/03 11:30 am EST -We saw a brief spike in transaction processing times between 10:50 and 11:00 am EST today. Impact is believed to be minimal. Things have stabilized again over the last 30 minutes. 

12/03 08:45 am EST -We show transactions processing within the usual timeframes after approximately 5:00 pm last night and continuing through this morning. We are still working to identify the root cause of the latency so we can address it permanently, but meanwhile we are operating normally today and closely monitoring processing times.

12/02 05:45 pm EST -We have seen processing speed reduce back to near normal levels over the last 45 minutes. We are continuing to monitor closely for any further indications of latency.

12/02 04:30 pm EST -We have determined that a small number of settlements also appear to be impacted by latency. Paya's Operations team already reviews incomplete settlements first thing every day, so we will identify any impacted settlements and address them within one day.

Our IT teams are taking action to help identify the cause of the latency. Users may experience extremely brief loss of connection to our gateway for the next few minutes.

12/02 02:40 pm EST -We are continuing our efforts to restore normal services. Periods of sporadic latency are likely to occur while our investigation is ongoing.

12/02 12:55 pm EST -We have completed our maintenance but the issue remains ongoing. We are working on next steps.

12/02 12:35 pm EST -Our IT teams are taking action that we expect to remedy the unusually long transaction times that some of our customers have experienced this morning.

12/02 12:20 pm EST -Our investigation remains active. We are committed to resolving the issue and returning processing to normal levels as quickly as possible.

12/02 10:50 am EST -Paya is currently investigating multiple reports of transactions processing very slowly or timing out through Paya Exchange integrated solutions. Customers attempting to make a purchase may receive an error or no response. Please check your Virtual Terminal reports prior to re-attempting any transactions that do not appear to complete. Our IT teams are engaged and working to resolve the issue as quickly as possible.