Postgres New Columns Not Showing

What we did:
We changed a few of our tables (dropped and renamed certain columns), which would break questions that were built on top of them. We were expecting to be able to amend the saved questions (either written in SQL or using the UI) and save questions with their new respective columns.

What we are seeing:
The new columns are not present in the GUI at all.

They are, however, there when you use the SQL engine: SELECT * FROM ....

This means we cannot at present fix any of the questions produced via the GUI.

What we have tried:
We have re-scanned the database and discarded the cache values, with no luck.

Any help or advice would be great!

Metabase v0.33.5

Try CTRL-F5 in the browser. There’s a lot that gets cached on the client for performance

No luck I am afraid @AndrewMBaines.

@Kieran_m
Upgrade to 0.33.5.1 if you are using the Docker version, since there were a build problem.
Just to make sure, are you also clicking “Sync database schema now” or just scanning? You need to sync.
Do you see any errors during the sync process? Admin > Troubleshooting > Logs

I will roll it up to 0.33.5.1 today and update whether that fixed it - Thanks @flamber

We are syncing as well, yes and no apparent error logs, just the error logs from the saved questions referencing fields that do not exist anymore.

The update didn’t work @flamber I am afraid.

We were seeing some memory issues and so provided the following config to our docker deployment ==> - name: JAVA_TOOL_OPTIONS value: '-Xmx496m'

That helped prevent the container falling over, but Metabase is still not correctly scanning the schema to capture the new fields.

I noticed this error from the logs:

[a5a4f484-da99-449a-8760-c440209c88c5] 2019-11-18T09:52:41+00:00 WARN metabase.middleware.log GET /api/database/10/metadata 403 3.3 ms (3 DB calls) "You don't have permissions to do that."

Database 10 is explorable using the Metabase UI so the database connection permissions are correct.

There is then a large error message:

[a5a4f484-da99-449a-8760-c440209c88c5] 2019-11-18T09:52:49+00:00 ERROR metabase.sync.util Error generating fingerprint for Field 23,785 'engines': class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap') ("clojure.lang.RT.doubleCast(RT.java:1355)" "bigml.histogram.core$when_double.invokeStatic(core.clj:104)" "bigml.histogram.core$when_double.invoke(core.clj:103)" ...

Then the column error appears as so:

[a5a4f484-da99-449a-8760-c440209c88c5] 2019-11-18T09:53:18+00:00 ERROR metabase.driver.sql-jdbc.execute Failed to set timezone: PSQLException: Message: ERROR: column table_name.Total GWP does not exist Position: 129 SQLState: 42703 Error Code: 0

I am not sure on why the exception is thrown on timezone setting - any ideas?

@Kieran_m
Dang it. I hoped it was just the bad 0.33.5 version.

As for the permission problem, it might be because of one of these, but I would need more information to be sure:
https://github.com/metabase/metabase/issues/11297
https://github.com/metabase/metabase/issues/11028

I’m fairly sure the fingerprint error is what is causing it not to show new columns.
I think you’re seeing this issue:
https://github.com/metabase/metabase/issues/10617

The timezone error. Have you redacted some of error or does it actually say column table_name.Total GWP does not exist?
Can you check your Postgres log to see the full query? It might help finding the root cause.

@flamber thanks for sharing the links, I have left comments :slight_smile:

No I haven’t redacted anything, the query is correct. Total GWP is a real column that was removed and replaced in the underlying source table. The error is what I would expect - albeit I don’t understand the context of the timezone element?!

But due to the possible fingerprint issue, I am unable to change the query to address the new column on the table. Rendering any affected questions built in the UI completely useless.

@Kieran_m
The timezone error might be two different queries, but it would really help if you checked the Postgres log, since it should show what queries are failing.
I think it’s trying to set timezone and then it’s also failing to find the non-existing column.

The UI breakage after changing the query, filter or database structure, is likely this issue:
https://github.com/metabase/metabase/issues/9299 - upvote by clicking :+1: on the first post

In case anyone reading this thread hits similar problems. A short-term fix to make Metabase usable again is to re-create the previously removed columns, with the old titles and blank ’ ’ entries.

This enables Metabase to properly sync with the new schema and bring in any new columns. Under the Admin panel you can hide these new “fake” columns from the UI. This is sub-optimal as your reporting layer now has obsolete code, but at least Metabase is useable again.

1 Like

Similar thread, check if you have any primary keys that are dates and if so change them to regular index. Error reported during Metabase sync task, sync not completing missing things like binning

You can also create a new database source and see if the new one syncs. If so, something has run amock, hopefully it’s as simple as this date primary key issue.