High rendering time of fields data in Field Filter


I have 28682 active fields in one of my database. The /database/{id}/fields api takes anywhere around 20-50 secs to fetch all the fields. And the rendering time of the fields in the Field Filter is also pretty high (~5-10 secs).

Does anyone else face this issue? Any pointers on what could be done? Other than fetching this in batches and adding pagination, of course.