Are the Spark.NET 3.5.1 works in Polyglot notebook for UDF examples?

[GeorgeS2019]

@grazy27
4. User-Defined Functions

Previously in older version, it is not possible to work in the UDF notebook. Not sure if the latest 3.5.1 overcomes this limitation

https://github.com/Apress/introducing-.net-for-apache-spark

[grazy27]

@GeorgeS2019 It does not.

It tries to get entire declaring type, which is anonymous assembly that only exists within interactive session context.
And fails with an exception, as result:

Code expects declaring assembly for UDF to be accessible:

Notebooks without UDFs and broadcasts work fine.

[GeorgeS2019]

Do U have recommendation to the polyglot team what changes they need to make to accommodate spark.net UDF to work?

I am going to invite them to address your findings

Good job

[grazy27]

Certainly, there are ways to achieve this.
In the context of Dotnet.Spark, we can determine whether the current code is running in an interactive environment and manually include library bytes in the command payload (or alternatively, save the library as a file and load it on the worker side).

PoC:

For this PoC to work, we also need to hardcode executing assembly as 1.dll

Polyglot team could do something similar on their side as well, but it'd have to be strongly names and uploaded to GAC, or explicitly target '%DOTNET_WORKER_DIR%'.
This would only work locally, and other dotnet.Interactive wrappers would need the same fix.

This issue falls primarily within Dotnet.Spark's scope of responsibility. This issue should be taken into account when addressing migration from BinarySerializer to somewhere else, since the responsible code will have to be rewritten.