Getting the Latent Space of a table

[42elenz]

Dear Team and Community,

I truly appreciate this project and the work being done here. I wanted to inquire if there’s an easy way to extract the latent space from this model. As someone fairly new to machine learning, I would greatly appreciate it if you could provide a code snippet or point me towards an API reference that could assist with this.

Thank you in advance for your help! :)

[AnFreTh]

Hi,

Thank you for starting this discussion and for your kind words about the project! Currently, none of the models in Mambular are pre-trained or prior-fitted on any tasks. While we do use embeddings, they are specific to the task and directly integrated into the model head. As a result, Mambular does not provide a way to generate embeddings for unseen data, unlike pre-trained models.

To clarify, are you looking to extract embeddings from a model you've already trained on your dataset, or are you asking whether we offer pre-trained models similar to TabPFN that can generalize to unseen tables?

[42elenz]

Just while training the model for unseen data.
My goal: I want to use the embedding for a contrastive task and update the weights of the model.
This would be a CLIP (Contrastive Learning) approach just using tabular data with images/MRIs. Thr encoders dont have to be pretrained. For this I would need a meaningfull embedding vector derieved from the tables. I am using an AE at the moment which is not the best option. And since your Tabular Mamba is outperforming even XGboost I was hoping to use your model :)

[AnFreTh]

Thanks for the clarification. We have not thought about this usecase so the library does not support it so far. Feel free to raise an issue with a fetaure request and we will add it in the next release (Hopefully within 4 weeks). In the meantime, you could look at the models in mambular/base_models and directly extract the latent spaces from there, but it would require to rewrite this base class which could be a bit tricky.

[42elenz]

I can try :) there should be an embedding before the prediction, so I think it shooould be possible to extract the latent space.
I will open a request.

[AnFreTh]

Yes it will be a bit annoying to do it manually since we did not consider this use case. If you come up with a solution feel free to contribute :). Otherwise we will try and include it in the next version update .