New Pamac search suggestion floods AUR search

Pamac Version

10.2.0

Libpamac Version

11.1.0

Desktop environment

KDE

What's not working

Searching in Pamac triggers suggestion, and floods the AUR, leading in non working AUR in Pamac (maybe even what caused the AUR crash today/yesterday)

[omano@omano-nvme ~]$ pamac-manager 

(pamac-manager:23212): Gdk-CRITICAL **: 18:35:58.792: gdk_window_set_cursor: assertion 'GDK_IS_WINDOW (window)' failed
Failed to query https://aur.archlinux.org/rpc/?v=5&type=suggest&arg=lib from AUR: Socket I/O timed out
Failed to query https://aur.archlinux.org/rpc/?v=5&type=suggest&arg=libr from AUR: Socket I/O timed out
Failed to query https://aur.archlinux.org/rpc/?v=5&type=suggest&arg=libre from AUR: Socket I/O timed out
Failed to query https://aur.archlinux.org/rpc/?v=5&type=suggest&arg=librew from AUR: Socket I/O timed out

Accessing the URL from web browser works.

See related threads:

Responsible use of AUR and Pamac AUR problem

How to reproduce?

• do some search in Pamac
• get blocked from searching/flooding the AUR

I suggest that search in Pamac doesn't flood the AUR, doesn't even search in the AUR by default (when AUR is enabled), clicking the AUR button could trigger the search if what we are looking for is not in the repositories, or if we want specifically to search in AUR).

Also as I didn't get a reply there, I will link a related issue, when the suggestion searches, Pamac hangs and is not responding anymore, this is getting worse with this AUR blocking.

I am also facing the same issue, pamac grinds to a halt after searching causing me to close it and opening again contributing to me doing more searches, disabling aur fixes the hang, maybe we could have some sort of search proxy/indexer in front of pamac handling all searches if we are keeping the autosuggest feature so pamac wont be throttled like it is now.

But what omano suggested is the best for now.

We have blocked pamac 11 for now on the AUR. This is the second time we had to resort to such methods due to combination of pamac itself making a lot of requests and the fact it has a lot of users too.

Pamac is no longer blocked. We are still monitoring the traffic and might block it again in case the load becomes unbearable.

Thanks for confirmation.

Actually the AUR search performs the same queries (for each character that is entered)

However, just get rid of that auto-suggest nonsense.
People have some clue what they'd like to install and I think they are willing to hit ENTER or click "Search" when they are done typing ;)

The reason why AUR gets flooded by pamac is because pamac sends a new search query with every keystroke.

Despite the behavior being similar on the AUR website, the big difference is that one user going to the AUR for searching for a specific package is different from ALL Pamac users having AUR enabled in Pamac searching for something (most people with AUR enabled for a single package and then use Pamac, will triger multiple searches in AUR even if they don't want to), when you multiply the number of Pamac users, which is probably a fairly high number, by the number of keystroke, it starts to pile numbers here, in a different way than if these users would search on the AUR website when needed.

I love to use the AUR from Pamac but this definitely needs to be looked at to avoid situation like this, it is only going to get worse with time, I imagine Pamac users do not decrease with time, more and more distro/users every month.

most people with AUR enabled for a single package and then use Pamac, will triger multiple searches in AUR even if they don't want to

That's the the thing.

-> Auto-suggest should be disabled (at least for AUR)
-> Only search AUR when explicitly selected (Maybe some separate search button or tab or whatnot; don't know how the UI looks like nowadays)

Maybe Pamac should have its own server to cache results and use that as a proxy of some kind

So, we did an investigation and, it seems that pamac was the trigger for a deeper issue with the way we were collecting logs via prometheus. So, a snowball effect happened which brought the AUR down. Pamac, and running on manjaro, is by far the biggest client on the AUR, judging by the User-Agent string. For a number of reasons the aurweb has some poorly designed SQL queries, and we don't have anyone working on it. A python FastAPI re-write is in the works, but even then, it would be nice if manjaro used their own infrastructure for everything, specially since pamac searches both on manjaro's own repos, as well as on the AUR.

I am not specifically proficient with php but I have worked a lot with SQL - so if @grazzolini could point to the relevant schemas and current queries - maybe I could be of assistance?

You can look at the current code base. But we're trying to move away from it. Regardless of that, it doesn't change the fact that manjaro could have an infrastructure of their own for this.

@grazzolini

Regardless of that, it doesn't change the fact that manjaro could have an infrastructure of their own for this.

This was discussed but a Manjaro team member claimed that it was pointless to do that, among the reasons implying that it would spend as much resources on your end syncing up a copy of the AUR than it would of ALL Manjaro users using it constantly. Would be nice if you supported them keeping a copy of the AUR for Manjaro users, that way Manjaro users aren't eating up resources

There is no need to copy anything, instead caching could/should be implemented. Having said that, I think the FastAPI implementation will solve a lot of issues. It would still be a good idea for manjaro to have their own caching.

It would still be a good idea for manjaro to have their own caching.

@grazzolini I have some thoughts on this - so to not pollute this closed issue - how may I contact you?

There is a somewhat easy way around this.

Download the list and use the list to display possible matches - only when any given package is selected - the info should be pulled from AUR

The list of packages scripts can be found - having a timestamp inside - at https://aur.archlinux.org/packages.gz

That would work to search the list of PKGBUILD, but this has been discussed previously and this list would be only good for that, and all the other information would still need to be fetched from the AUR website anyway (but that would relief the AUR website for auto suggestion that's for sure).

We might have something that would augment the packages.gz file coming soon. That way a tool like pamac (and other tools) could download that and parse locally for auto suggestion. There are several causes for this issue: pamac sending one request per letter, huge number of pamac users (by far on manjaro), some AUR queries being poorly designed (there are some that trigger a full table scan), and so on. If you sum all that, you get a platform that can, in some peaks of traffic, get so jammed it stops responding.

It sounds like aur doesn't have package.gz. The cached package.gz locally has always been very useful. This has been practiced on pacman.

Using only local cache for autosuggestion would make sense. It should improve the performance also for pamac users. Cache could be refreshed when user presses enter to search or when pamac gui is loaded.

The size of this file is huge, so far, it's 4M compressed, 12M uncompressed. So fetching it all the time is not doable.

Hmmm... Good point. With pacli we store it in /tmp and fetch when the file is missing, so it happens once per boot. Maybe a systemd timer to fetch it once a day? AUR doesn't change so fast that it would need to be more often. But then it should not be a specific time of the day, so that all pamac users would initiate a download at the same time, as that might be bad for the server too...

How about downloading it when checking for updates?

This is still a draft. So, the idea is to write this file only if it has changed. And then pamac (and others) can use the Last-Modified header to download it or not.

pkglists on the aur are generated every 5 minutes. But this might change.

We did a research on how big is pamac impact on the AUR. Well, this MD give some insights. As one can see, pamac, running on manjaro is by far (more than double the second UA), the biggest AUR user. So, it will probably be the first thing we might block, to mitigate issues, in situations like the one that happened yesterday. It certainly doesn't help pamac sends one request per keystroke, even if a package might not even be on the AUR, by the time the user finished typing.

We have disabled the broken metrics on the AUR, and we're working on a solution that will include all the fields listed here on a pkglist.

In the MD you point that the larger part of pamac queries use the search API. Pamac sends one request per keystroke only with the suggest API. I implemented it at your suggestion, copying the web interface behavior. I can remove the usage of the suggest API but I don't know if it would really help.

@grazzolini thank you for the research and RAW data posted!

As one can see, pamac, running on manjaro is by far (more than double the second UA), the biggest AUR user.

If that users are those who sends a search request only and not those who installed packages via AUR, then yes. Probably you meant AUR's API user, not a users of the actual AUR scripts to install packages. OK.

---

pamac, running on manjaro probably be the first thing we might block, to mitigate issues

Pamac has the same behavior on all platforms (every-name-OS), right?

One of excessive usage as was noted by @omano:

most people with AUR enabled for a single package and then use Pamac, will trigger multiple searches in AUR even if they don't want to

and you:

pamac sends one request per keystroke, even if a package might not even be on the AUR, by the time the user finished typing.

yes, for example with idea to search local repo package or installed one while doing a search in default All section, which includes search in AUR and which triggers the AUR's API requests also.

So to mitigate the issue of high AUR's API usage, pamac as user agent should be blocked, not only Manjaro origin/auditory specifically. If to block pamac only on Manjaro, you "killing" only Manjaro auditory, not the issue itself: AUR's API will get still the same kind excessive API requests from other platforms which using pamac.

What I am about: if you want to prevent issue, than block all pamac. Blocking only pamac requests originated from Manjaro auditory is an action only against Manjaro users, not the root/source of issue origin.

Of course I do not want pamac to be blocked on any OS, but I am about your choice selection in the hope-never-be-the-case of blocking: to cut the issue source (software instrument of excessive requests) OR to cut exactly certain OS users only, but it is not the OS-related issue.

---

Now about possible fix:

What if to save all in pamac functionality as was a few days ago before this issue, with all auto-complete/suggestion drop-down lists, etc., but to fix it the next way.

Was (default is All section):

Suggested (default is the same section, but it is without AUR):

or with any other name:

so the idea to exclude search in AUR from All section, selected by default in the pamac.

By implementing this we:

1. cuts off excessive AUR's API usage while users wants to find something in local repo or as installed package;
2. all comfortable auto-suggestion features will work in pamac in every category, including AUR;
3. only after manual AUR section there will be all auto-complete API requests to AUR service and they will be just the same as https://aur.archlinux.org/ has, not less search requests on every keystroke, and not more, so it is pure shrinking of users count without excessive requests: just the same as a user would search on the https://aur.archlinux.org/ web-page directly.

@grazzolini , am I wrong or that could be a possible fix also?

Thanks!

libpamac 11.1.1 now doesn't use anymore the AUR suggest API

added High Prio help wanted labels