ALERT: RESOLVED
My Virtual Reports - Batch Summaries for Transactions Submitted 11/30/2019 Unavailable
12/04/2019 - 12/06/2019

12/09 08:00 am EST -Paya has uploaded the settlement files for transactions settled on 11/30/2019 to My Virtual Reports as of 12:00 pm EST on Friday, 12/06/2019.

12/04 09:45 am EST -Paya has identified that the settlement files for transactions submitted on 11/30/2019 may not be displayed under Daily Batch Summaries in My Virtual Reports for some of our customers. These files were received and processed; this is an issue in reporting only. We are working to import the missing settlements as quickly as possible.

ALERT:
Payment/Vault/Settlement Transaction Latency/Timeout in Paya Gateway
12/02/2019 - 12/06/2019

12/06 04:40 pm EST -A code deployment made this afternoon did not work as anticipated. We saw a brief spike in processing times following that deployment. The code has been rolled back and latency has returned to normal.

12/06 10:00 am EST -We deployed fixes on the evening of 12/05. By 09:00 pm EST we were no longer experiencing spikes in latency. That has remained true over the last 13 hours. We are continuing to monitor our systems closely to ensure we do not see any new rise processing times.

12/05 03:55 pm EST -Our fixes are still in testing. While that testing continues, we have automated our process of monitoring and correcting latency spikes as they develop. That process was taking approximately 15 minutes per occurrence and is now taking approximately 2 minutes.

12/05 12:00 pm EST -We have identified three sources in our code base that are contributing to the sporadic latency our customers have been experiencing over the last several days. Our IT teams are engaged in building and testing fixes to address those areas and return processing speeds to normal.

 We expect to have  fixes tested today. We will provide an update regarding implementation timing at 3:30 pm EST.

Please contact our Support team at support@paya.com for assistance in reversing any payments that may have been processed more than one time as a result of latency or timeout errors.

12/04 04:30 pm EST -Our investigation to identify the cause of sporadic latency remains ongoing. Our teams are continuing to monitor for occurrences of latency and take corrective action as they happen.

12/04 10:50 am EST -Our investigation to identify the cause of sporadic latency remains ongoing. Our teams are continuing to monitor for occurrences of latency and take corrective action as they happen.

12/03 05:05 pm EST -The source of the latency hasn’t been determined. We’re doing everything we can to find the source and remedy it. We are able to identify the spikes in latency as they are happening and we have teams monitoring our systems 24/7 specifically for these occurrences. Since we cannot predict when a spike will happen, we will always be reactive, and because of that spikes will result in some transactions taking a long time to process or timing out. Timeouts will sometimes lead to customers or consumers retrying transactions, resulting in duplicate payments. Our Support teams can assist in identifying and remedying any duplicate payments.

12/03 02:55 pm EST -We are currently experiencing another spike in transaction processing times that began at approximately 02:28 pm EST.

12/03 01:05 pm EST -We experienced another spike in transaction processing times between 12:22 and 12:52 EST today. Our teams are continuing work to identify the root cause and provide a resolution as quickly as possible.

12/03 12:25 pm EST -Numerous customers have reported transactions submitted more than once as a result of latency issues yesterday and during a brief period this morning. Please check your Virtual Terminal reports prior to re-attempting any transactions that do not appear to complete.

12/03 11:30 am EST -We saw a brief spike in transaction processing times between 10:50 and 11:00 am EST today. Impact is believed to be minimal. Things have stabilized again over the last 30 minutes. 

12/03 08:45 am EST -We show transactions processing within the usual timeframes after approximately 5:00 pm last night and continuing through this morning. We are still working to identify the root cause of the latency so we can address it permanently, but meanwhile we are operating normally today and closely monitoring processing times.

12/02 05:45 pm EST -We have seen processing speed reduce back to near normal levels over the last 45 minutes. We are continuing to monitor closely for any further indications of latency.

12/02 04:30 pm EST -We have determined that a small number of settlements also appear to be impacted by latency. Paya's Operations team already reviews incomplete settlements first thing every day, so we will identify any impacted settlements and address them within one day.

Our IT teams are taking action to help identify the cause of the latency. Users may experience extremely brief loss of connection to our gateway for the next few minutes.

12/02 02:40 pm EST -We are continuing our efforts to restore normal services. Periods of sporadic latency are likely to occur while our investigation is ongoing.

12/02 12:55 pm EST -We have completed our maintenance but the issue remains ongoing. We are working on next steps.

12/02 12:35 pm EST -Our IT teams are taking action that we expect to remedy the unusually long transaction times that some of our customers have experienced this morning.

12/02 12:20 pm EST -Our investigation remains active. We are committed to resolving the issue and returning processing to normal levels as quickly as possible.

12/02 10:50 am EST -Paya is currently investigating multiple reports of transactions processing very slowly or timing out through Paya Exchange integrated solutions. Customers attempting to make a purchase may receive an error or no response. Please check your Virtual Terminal reports prior to re-attempting any transactions that do not appear to complete. Our IT teams are engaged and working to resolve the issue as quickly as possible.