tag:poshtechnologies.statuspage.io,2005:/historyPosh Technologies Status - Incident History2024-03-21T16:43:16-04:00Posh Technologiestag:poshtechnologies.statuspage.io,2005:Incident/203225162024-03-21T16:26:12-04:002024-03-21T16:26:12-04:00Data Issue in Insights and Chat Logs in Posh Portal<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:26</var> EDT</small><br><strong>Resolved</strong> - The data issue has been resolved and data has been backfilled in the Insights and Chat Logs page of the Portal.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:04</var> EDT</small><br><strong>Monitoring</strong> - We experienced data pipeline issues in the Posh Portal Insights and Chat Logs pages this afternoon. Data from ~2:30pm ET to ~4pm ET was not lost, but is not populating in the Posh Portal yet. We have fixed the data pipeline issue so new data should now be populating in the Posh Portal. Data from 2:35pm ET to 4pm ET was not lost and will be backfilled tonight and available tomorrow.</p>tag:poshtechnologies.statuspage.io,2005:Incident/202159812024-03-11T11:46:19-04:002024-03-11T11:46:19-04:00Knowledge Assistant Search Slowness and Analytics Page Down<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:46</var> EDT</small><br><strong>Resolved</strong> - We have not seen recurrence of the issue. Searches seem to be operating with normal response times now and the Analytics page is responding normally too. We are resolving this incident.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:31</var> EDT</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring performance. Searches look to be running faster and the Analytics page is now loading.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:08</var> EDT</small><br><strong>Investigating</strong> - Knowledge Assistant searches are currently operating more slowly than normal and the Analytics page isn't loading. We are investigating. No other Assistants (Digital or Voice) are impacted.</p>tag:poshtechnologies.statuspage.io,2005:Incident/201069592024-02-29T21:46:26-05:002024-02-29T21:46:26-05:00Voice Assistants Degraded Performance<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>21:46</var> EST</small><br><strong>Resolved</strong> - We haven't seen any further issues - last issue we observed was 3 hours ago. We are resolving this incident now, and we will continue working with Twilio to better understand the cause and how to prevent reoccurrence.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>19:43</var> EST</small><br><strong>Monitoring</strong> - We have seen very few errors in the last 40 minutes. We are continuing to monitor the issue and work with Twilio.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>17:23</var> EST</small><br><strong>Update</strong> - We tested a fix earlier, but unfortunately that did not resolve the issue. Posh is actively engaged with Twilio to continue investigating and will be throughout the evening. We will continue to provide updates as information becomes available.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>14:00</var> EST</small><br><strong>Update</strong> - Twilio continues to investigate and has provided Posh a potential workaround that we are actively implementing.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>12:10</var> EST</small><br><strong>Update</strong> - Twilio has escalated this issue internally, we continue to actively engage with Twilio and expect another update from their engineering team in approximately 30 minutes.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:56</var> EST</small><br><strong>Update</strong> - We have been in active conversation with Twilio for 20 minutes. Still investigating a fix.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>09:10</var> EST</small><br><strong>Identified</strong> - The issue has been identified and we are awaiting support from our partner, Twilio. We know they are investigating as they've confirmed that with us, and we should hear from them again in ~20 minutes with an update.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>08:19</var> EST</small><br><strong>Investigating</strong> - Voice Assistants are experiencing issues where the bot is cutting itself off without voice input from the caller, particularly in the intro message (start intent). This is not happening on every call nor for every bot, but many are impacted. We are investigating this.</p>tag:poshtechnologies.statuspage.io,2005:Incident/201080352024-02-29T14:10:13-05:002024-02-29T14:10:13-05:00Posh Portal Content Studio is Down<p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>14:10</var> EST</small><br><strong>Resolved</strong> - No further issues have been found with the Content Studio. We are closing this incident.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>11:59</var> EST</small><br><strong>Monitoring</strong> - A fix to the Content Studio has been implemented, so the product is now functioning again. We are still monitoring performance for any reoccurrence of issues.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>10:27</var> EST</small><br><strong>Investigating</strong> - We are currently investigating issues with the Posh Portal Content Studio, as it is not functioning as expected.</p>tag:poshtechnologies.statuspage.io,2005:Incident/200893012024-02-27T13:27:25-05:002024-02-27T13:27:25-05:00Posh Portal Content Studio is Down - Bots are NOT Impacted<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>13:27</var> EST</small><br><strong>Resolved</strong> - No further issues with the Content Studio have been seen, so we are closing this incident.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>12:50</var> EST</small><br><strong>Monitoring</strong> - Content Studio is now functioning as expected. We are monitoring for any additional issues, but believe this has been rectified.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>11:52</var> EST</small><br><strong>Identified</strong> - We are working on a fix to the Content Studio, and some functionality has already returned.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>11:33</var> EST</small><br><strong>Investigating</strong> - The Content Studio page of the Posh Portal is experiencing issues. We are investigating. No bots are impacted - all bots are functioning normally.</p>tag:poshtechnologies.statuspage.io,2005:Incident/196589102024-01-12T03:00:24-05:002024-01-12T03:00:24-05:00Posh Portal: Insights & Chat Logs Maintenance<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>03:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>23:01</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>12:40</var> EST</small><br><strong>Scheduled</strong> - We will be performing routine maintenance on the Posh Portal on Thursday 1/11/24 overnight from 11pm Eastern on 1/11 to 3am Eastern on 1/12. The only pages impacted will be Insights and Chat Logs. All other pages (Coverage Gaps, Content Studio, Bot Configuration, Users) will not be impacted. If you do go to the Insights or Chat Logs pages during maintenance, data will not load on the page. <br /><br />As this only impacts the Posh Portal, all Assistants will be fully operational during this maintenance period.<br /><br />We will send notifications 1 hour before as a reminder, once maintenance starts, and then once it is completed.</p>tag:poshtechnologies.statuspage.io,2005:Incident/196056372024-01-03T16:22:13-05:002024-01-03T16:22:13-05:00Posh Portal Content Studio: Trouble Loading Intents<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>16:22</var> EST</small><br><strong>Resolved</strong> - The Posh Portal Content Studio is now functioning as expected again.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>16:19</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are continuing to monitor.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>16:05</var> EST</small><br><strong>Investigating</strong> - We are currently investigating an issue with the Content Studio page of the Posh Portal with intents not loading properly. Bot performance is NOT impacted.</p>tag:poshtechnologies.statuspage.io,2005:Incident/194049462023-12-11T16:43:15-05:002023-12-11T17:00:29-05:00Knowledge Assistant Searches are Down<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:43</var> EST</small><br><strong>Resolved</strong> - Knowledge Assistant is now functioning normally again.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:07</var> EST</small><br><strong>Investigating</strong> - We are currently investigating an issue being able to perform a search with Knowledge Assistant. This does not impact any member/customer-facing Digital or Voice products.</p>tag:poshtechnologies.statuspage.io,2005:Incident/193369652023-12-05T13:09:39-05:002023-12-05T13:35:14-05:00Knowledge Assistant incident<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>13:09</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved and Knowledge Assistant bots should now be functioning as expected.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>12:35</var> EST</small><br><strong>Investigating</strong> - We are currently investigating an outage with the Knowledge Assistant. This does NOT impact any member/customer facing products (digital or voice).</p>tag:poshtechnologies.statuspage.io,2005:Incident/190590362023-11-08T11:05:21-05:002023-11-08T11:05:21-05:00Knowledge Assistant and Poshie<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:05</var> EST</small><br><strong>Resolved</strong> - Our provider has fully resolved their incident and Knowledge Assistants and Poshie have not experienced issues for the last 20 minutes since our last update. This incident is now resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:44</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented with our underlying provider. We are seeing recovery and are still monitoring Knowledge Assistant and Poshie products.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:40</var> EST</small><br><strong>Investigating</strong> - Poshie and the Knowledge Assistant products are experiencing an outage (Digital Assistants and Voice Assistants are NOT impacted). Search functionality is currently down in Poshie and Knowledge Assistant because one of our underlying providers is having an outage.</p>tag:poshtechnologies.statuspage.io,2005:Incident/188690062023-10-20T13:28:43-04:002023-10-20T13:28:43-04:00Chat Logs in Posh Portal is Down<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>13:28</var> EDT</small><br><strong>Resolved</strong> - Chat Logs in the Posh Portal is now fully functional. The incident stemmed from an engineering error that caused the underlying database to lock. Posh will be reviewing database-related practices to prevent further incidents.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>12:51</var> EDT</small><br><strong>Investigating</strong> - The Chat Logs page in the Posh Portal is currently not responding. We are investigating and will keep you updated. Please note that all bots are functional - this is not impacting members/customers.</p>tag:poshtechnologies.statuspage.io,2005:Incident/186700992023-10-02T17:48:39-04:002023-10-02T17:48:39-04:00Voice Bot Voice Input Recognition & Glia Digital Bot Issue<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:48</var> EDT</small><br><strong>Resolved</strong> - Google just updated their status to reflect that this issue has been resolved on their end with Speech to Text. So Posh bots should not experience any further issues.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:45</var> EDT</small><br><strong>Update</strong> - This issue with voice bots resulted from a Google Speech to Text outage that Twilio (Posh's telephony provider) and Posh relies on. Google has shared that they expect this service to be fully restored by 6:10pm ET. We have been monitoring and haven't seen any issues in the last 30 minutes. We will be monitoring this incident until Google announces full service restoration.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:33</var> EDT</small><br><strong>Update</strong> - We have seen stable voice bot performance for the last 10 minutes. We are continuing to monitor.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:00</var> EDT</small><br><strong>Monitoring</strong> - Glia-integrated digital bots seem to be working again now. Glia has updated their status page that they are investigating an issue on their end: https://status.glia.com/ We will be continuing to monitor. <br /><br />Voice is still being monitored and is intermittent - Twilio's status page hasn't yet been updated, but we are seeing fewer errors across bots. We are continuing to monitor.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>16:40</var> EDT</small><br><strong>Update</strong> - The only digital bots impacted by this issue are Glia-integrated bots. We are continuing to investigate this issue.<br /><br />Twilio has updated their status page that they are having issues that are impacting Posh. We are continuing to monitor Twilio's status.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>16:29</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>16:19</var> EDT</small><br><strong>Update</strong> - Some digital bots are also impacted by this outage. We are continuing to investigate.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>15:58</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with voice bots recognizing voice input.</p>tag:poshtechnologies.statuspage.io,2005:Incident/164027432023-03-09T10:35:01-05:002023-03-09T10:35:01-05:00Posh bots with downtime this morning<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>10:35</var> EST</small><br><strong>Resolved</strong> - Posh bots experienced some downtime today 3/9/23 from 9:51 to 10:06am ET. Each bot was impacted, but not for the entirety of that time period. As of 10:06am ET, all products are functional.</p>tag:poshtechnologies.statuspage.io,2005:Incident/126968172022-11-02T11:00:00-04:002022-11-02T13:28:01-04:00Posh Product Infrastructure Downtime<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>11:00</var> EDT</small><br><strong>Resolved</strong> - On 11/2/22 from 11AM to 11:04AM, Posh infrastructure experienced an outage. This impacted all products. This issue occurred during a normal release to upgrade database infrastructure. Posh identified the issue immediately and took quick action to recover to minimize downtime.</p>tag:poshtechnologies.statuspage.io,2005:Incident/107920582022-08-09T15:40:38-04:002022-08-09T15:40:39-04:00Intent recognition issue<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>15:40</var> EDT</small><br><strong>Resolved</strong> - Intent recognition was brought back online at 3:33pm ET. All products are back to operational.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>15:31</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>15:28</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with intent recognition across all the Posh products.</p>tag:poshtechnologies.statuspage.io,2005:Incident/106880902022-07-25T14:54:46-04:002022-07-25T14:54:46-04:00Bots using new Posh CMS are down<p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>14:54</var> EDT</small><br><strong>Resolved</strong> - We have fixed the issue and all bots are back online and functioning properly. Incident began at 1:39pm ET and ended 2:51pm ET.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>13:39</var> EDT</small><br><strong>Investigating</strong> - Any bot that uses the new Posh CMS isn't functioning properly right now. We are currently investigating and will report back once we have an ETA on fixing the issue.</p>tag:poshtechnologies.statuspage.io,2005:Incident/96497792022-03-28T12:43:54-04:002022-03-28T12:43:54-04:00Twilio experiencing degraded performance<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:43</var> EDT</small><br><strong>Resolved</strong> - We have successfully been operating the Voice products since 11:45am ET, so we are marking this incident now as resolved.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:11</var> EDT</small><br><strong>Update</strong> - Performance seems to have recovered. Twilio is still monitoring, so we will continue monitoring as well.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:50</var> EDT</small><br><strong>Monitoring</strong> - Twilio is reporting degraded performance on their status page: https://status.twilio.com/<br />They have just started reporting this, but the impact seems to be minimal right now. We are continuing to monitor.</p>tag:poshtechnologies.statuspage.io,2005:Incident/95737602022-03-18T11:26:28-04:002022-03-18T11:26:28-04:00Major Outage<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>11:26</var> EDT</small><br><strong>Resolved</strong> - All bots are now functional and this incident is resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:45</var> EDT</small><br><strong>Update</strong> - All website bots are now back up and fully operational. We are triaging some issues we are still seeing with voice products.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>10:10</var> EDT</small><br><strong>Monitoring</strong> - A fix is now in place and bots are responding but still a bit slow. We are continuing to monitor and will update once fully resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:32</var> EDT</small><br><strong>Update</strong> - Should be another 10 minutes of downtime for voice and website bots. Banking bots will take a bit longer to get back up and running.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>09:10</var> EDT</small><br><strong>Update</strong> - We are continuing to work on a fix to this issue. ETA for resolution is unknown at this time.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>08:49</var> EDT</small><br><strong>Identified</strong> - One of the Posh servers crashed. We are identifying the source and rebuilding the server. Expectation is another 10-15 minutes of downtime.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>08:28</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>08:27</var> EDT</small><br><strong>Investigating</strong> - We are investigating a current outage, and will keep you updated. All bots seem to be impacted.</p>tag:poshtechnologies.statuspage.io,2005:Incident/86171082021-11-16T15:13:00-05:002021-11-19T15:16:24-05:00Posh Full Outage - Google Cloud Platform<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:13</var> EST</small><br><strong>Resolved</strong> - Posh experienced an outage with our underlying cloud platform provider, Google Cloud. This incident was widespread, resulting in 404 errors to GCP-hosted services and applications. This affected other major GCP-hosted applications, including Snapchat, Etsy, Spotify, Home Depot etc. The total outage occurred from 12:52 - 1:10pm on 11/16/21.</p>tag:poshtechnologies.statuspage.io,2005:Incident/86179812021-11-10T03:30:00-05:002021-11-19T19:37:23-05:00Partial Outage<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>03:30</var> EST</small><br><strong>Resolved</strong> - A subset of Posh web and ivr bots were impacted by a partial outage due to an underlying infrastructure issue. Minor performance issues for this subset of bots were seen from 3:30 - 8am, and significantly degraded performance from 8am - 9:30am.</p>tag:poshtechnologies.statuspage.io,2005:Incident/86178122021-10-15T17:30:00-04:002021-11-19T19:37:49-05:00IVR Bot Outage<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>17:30</var> EDT</small><br><strong>Resolved</strong> - Today from 12:35 - 1:20pm EDT, Posh IVR bots experienced an unexpected outage where some calls were able to get through but most calls during this time were experiencing issues.</p>