Item and category changes not syncing to Square Online

[keeprefillery]

Since about a week ago, we've been unable to make category and item updates on one of our Square Online channels.

Now the only significant change we'd made was to enable meta sync-ing of products (which is terrible, borderline useless but that's a different issue)... We have disabled the meta sync, we've tried bulk updating, individual updates, toggling items hidden/visible, removing them from the channel and re-adding them, adding and removing them from categories, creating new categories, creating new items... nothing seems to sync to Square Online.

We've been waiting three days for Square Support to answer us, but since they still haven't and this is quite damaging we were hoping perhaps someone in the community might have had a similar instance before, and even a solution?

Thanks in advance, any insights would be appreciated.

P.S. Sorry if this ends up duplicated, but I posted it once and I couldn't find it since I was stopped to input a username and then my post seems to have been lost, so posting again... (bugs, bugs everywhere!)

[keeprefillery]

Square contact said "unavailable", but they actually were - we've spoken to them, they reindexed the items, and the website updated. The issue however persists at the moment as further changes are still not synchronising, seems it's very much a behind the scenes tech issue so we'll keep chasing it up, if anyone else encounters it I suggest you do the same.

[Sineaid]

Hi @keeprefillery,

Welcome to the Community, and sorry to see your first post is under these circumstances. 

From what you've said, something certainly doesn't sound quite right. I've scoured our internal channels to see if any other Sellers are potentially experiencing Square Online sync issues after connecting Meta for business, however I've not seen any other reports around the time of your post.

If you're still having trouble with this, would you mind sharing a link to your site so I can look into it further? 

I'll keep an eye out for your response.

[keeprefillery]

Hey Sineaid, sorry I could have sworn I posted a reply already - we are still having this, it's actually impacting both of our sites so they're both currently useless. The sites are thekeeprefillery.co.uk, and blossomandswan.co.uk

So far nothing useful from support, it's supposed to be getting looked into properly but nothing as of yet and it's been over a week since it was escalated to the engineering team.

[Sineaid]

Thanks for letting me know you're still having trouble with this @keeprefillery, and sorry to hear your experience with support has been disappointing. 

I'll go ahead and pass this across to our eCom engineers, and let them know the issue persists despite the previous attempt to resolve by re-indexing.  

I anticipate our team will request an example of a recently updated product or category that has failed to sync, so I'd appreciate it if you could drop a few links/screenshots here, that I could share.

What would be even better, is if you could try again to update a product or category, so that we also have an approximate time stamp we can share with our engineers. 

I'll let you know here either way, as soon as I have an update.  

[keeprefillery]

Thanks @Sineaid, just so you're aware, this was escalated to engineering on Feb 10th by the ecom success team, we got an update on Feb 19th saying a fix had been made and we're to wait 1-2 hours, then publish an update and try again.

That made no difference, left it overnight, still nothing. We since made some item updates which have yet to show up, just now (at 18:12) I added the word "TESTING" to the title of one of our products:

https://www.thekeeprefillery.co.uk/product/bodywash-bubble-bath-miniml-/1408

I am currently waiting before I post this, but so far no change in the last five minutes.

So at this point both of our websites have been essentially useless for over a month 😐 and progress has been slow.

[Sineaid]

Thanks for the update, and for providing the link, @keeprefillery.

I've supplied all the information to our Engineers, and will let you know as soon as I hear back.

[keeprefillery]

So just to summarise where we're at...

Weebly support haven't replied to us when we provided details back on the 7th of Feb, they asked for the info, we gave it them and then nothing since.

Square support have been ignoring our emails for the last two weeks since they apparently "fixed" the issue.

I just got an auto reply saying you guys haven't heard from us so you've closed the case which is nonsense.

And on this front it's been over another week with no progress, meanwhile both our sites remain broken and useless, and we're a tad frustrated.

[Sineaid]

Hi @keeprefillery,

Thanks for the update, and for your continued patience.

It's disappointing to hear about your experience with support, it's not what we expect for our Sellers when they reach out for help, so I fully understand your frustration here.

While I wouldn't be able to look into your interaction with support, I have reached out to our Engineers again this morning to press for an update. 

I'll let you know as soon as I have a response. 

[Sineaid]

Hi @keeprefillery,

I've had an update from our Engineers to advise they had identified a problem with your location settings, which seemed to be causing the issue.

The location settings have now been amended, and our team is confident that the issue is resolved. 

If you do experience any other unexpected behaviour, or discrepancies with your item sync, please just let me know and I'll push this back to our engineers as a matter of priority. 

[keeprefillery]

Hey @Sineaid Still no change at all. I have tried again to update the title of the item I've been using as a test:

https://www.thekeeprefillery.co.uk/product/bodywash-bubble-bath-miniml-/1408

Item title remains "Bodywash + Bubble Bath (Miniml)" where as in Square the title is "Bodywash + Bubble Bath (Miniml) TESTing", it has been well past 5 minutes again since I made the change and waited for the sync to occur. I also tried another item just for good measure, same result, no change.

Can I ask that engineering actually test the issue? They are welcome to go to the item in our Square dashboard and remove the word "TESTing" from the end to verify the sync. That's the second time they've been confident they've fixed it, and we're still here 😄

[keeprefillery]

Plus there are a bunch of the newer products we'd been adding when this issue first occurred which are still not present on the website at all, IF there fix has actually worked, it might be the site also needs a full catalogue resync to bring everything up to date.

In any case it would be nice if they actually tried and verified the issue after "fixing" it, if it helps I have added a dummy item completely for testing purposes:

https://app.squareup.com/dashboard/items/library/6HKZC6QEAAGW4GHHQEMOKOTD

that should appear in our Local Refills category:

https://www.thekeeprefillery.co.uk/shop/local-refills/7HYVFV7X6N46Q3SXYIMBEJWT

but doesn't, it's not a real item though so they can feel free to update/play with it as needed to QA the issue.

[Sineaid]

Hi @keeprefillery,

Thanks for the response.

Our Engineers confirmed that they did test the fix by adding and removing the word 'Test' to several items. They report that the sync was effective on your live site within just a few minutes. 

Would you mind heading into the Item editing screen for the product: Bodywash + Bubble Bath (Miniml) TESTing

Once there could you scroll down to the Site Visibility section, and ensure Visible is selected.

If you don't see the item displayed after doing this, could I ask that you republish your site from your Site Editor.

Let me know how you get on. I'll be more than happy to go back across to our eCom engineers if you're still not seeing the expected result. 

[keeprefillery]

Thanks, I have tried toggling the item visibility, no change.

I have tried unpublishing and re-publishing the site, no change.

The test item I added above is not visible at all still, and the item I've been changing the title of still remains unchanged:

but remains visible on the site:

and these items we added when the issue started:

They are all in the correct channel, all "Visible", and still none of them (same as the test item) are showing on the site at all, even if I go to "All Items" instead of the "Local Refills", still no sign of them.

For paranoia sake I have confirmed the "square_id" in the API payload matches that of the product I am updating in Square.

it has now been 20 minutes since I made the changes and re-published, and none of the updates are showing

[Sineaid]

Thanks for the comprehensive reply @keeprefillery!

I'll take all of the above across to our Engineers, and ask that they investigate further. 

I'll let you know as soon as I have an update. 

[keeprefillery]

Hey @Sineaid anything at all yet? Sorry to nag but it's been another two weeks, in a week's time our websites will have been useless for two whole months...

[Sineaid]

Hi @keeprefillery,

It's no trouble at all, and certainly not nagging! 

I've been tracking your ticket with our engineers, and while they have identified that the root cause is rate limit error due to a high number of variations, they have established that it is part of a wider issue that's currently being worked on. 

I've reached out to the lead engineer to see if they can provide an ETA for the deployment of the fix. I'll let you know as soon as I have a response. 

[keeprefillery]

Hey @Sineaid,

So we heard back from Square earlier by email basically saying the same thing you said a week ago, and that our "intermittent" (its persistent) issue is due to our 89,000 variants, etc. rate limiting, etc.

Now while the notion of this being rate limiting seems comically bad, the mention of us apparently having 89,000 variants struck me as being highly unlikely, we started using Square a year ago, it should immediately flag as suspicious that we could have added that many items in that time (250 new variants per day?).

So I've just been making a few (hundred or so) API requests, and while I didn't go through the entire catalogue I can see there are a indeed ridiculous number of them, the vast majority of which appear to be duplicates that we cannot account for.

Starting from `2025-01-05` we suddenly see a large number of new catalogue items and variants being created seconds apart from each other and constantly.

First instance appears to be Item #TRMZCWFQKELGMQV7SZ7VH4FD at 2025-01-05T21:59:38.287Z. Up until then we had approximately 2,300 variants, by 2025-01-05T23:59:46.337Z we had 24,700 variants and it keeps going...

I had a look at some of the duplicate items, they appear uncategorised, they have no stock or images at least on the few I checked, and when viewing one the system flags up that the SKUs are duplicated.

So I'm not sure if this coincides with our attempt to enable the meta sync which appeared to get itself stuck in a loop, but either way, the system has clearly done something odd, and generated a load of duplicate items.

Can you please take this back to the engineering team and get them to have a proper look at and fix the data, as that is clearly where the actual issue is, I would hope you have more audits/logs on your end to see where these records came from but either way, if you could clear them out and re-sync the catalogue it sounds like our problem will go away. Thanks.

[keeprefillery]

We're approaching yet another week gone by while we're offline since January, even after I've done all the debugging work for you guys, this is beyond a joke at this point.

We did have a reply from Square support by email again, they basically told us to fix it ourselves, and didn't even bother to look into it:

    "we don’t generate duplicates on our end"

...they said, so it must have been us that added 85,000+ items in the space of a day, how silly of us.

[keeprefillery · Accepted solution]

So for the record:

• Square were repeatedly unable to locate the issue
• Wasted loads of our time jumping through the same hoops over and over
• Made two supposed fixes which did nothing
• More then two months later; Misidentified it as rate limiting problems which we can demonstrate are due to some issue on their end replicating some items in our library thousands of times
• We pointed out the data issue they'd overlooked, they refused to fix it and blamed us for it
• Tried to at least get them to look into it, they can't, they don't keep logs that far back
• Our websites still don't work
• We still have 89k variants to cleanup manually thanks to the bug

we're bordering on three months without our websites now, and we've given up; we'll fix the data ourselves and stop using Square/Weebly for our websites, this level of incompetence is too risky to work with.