Preparing Apache Kafka for Scala 3

A Story in 3 acts

1. Origin
2. Challenges during the PoC
3. What should we improve?

Origin

Who?

Why?

What?

Who?

Josep Prat

Might have seen me around Akka.

Working at Aiven.
Manager of the Open Source Program Office.

Why?

Aiven offers Managed Open Source Data Infrastructure as a Service.

And many more…

Why?

The Open Source Program Office employs people to work full time on Open Source projects.

What?

Apache Kafka is:

• Distributed event streaming platform
• Extremely scalable
• High throughput
• High availability

What?

Some internal details on Apache Kafka:

• Compiles against Scala 2.12 and 2.13
• Uses Gradle as a build tool
• Kafka Core is mostly written in Scala
• No macros nor typeclasses

What?

Seemed like a walk in the park…

…or not!

Challenges during the PoC

Gradle

How complicated could it be to compile using the Scala 3 compiler instead?

Gradle

One might think:

• Change dependencies.gradle
• Change build.gradle
• Do some magic with version names
• Profit

Easy, right?

I was wrong…

Big kudos to Tomasz Godzik!

For singlehandedly introducing support for Scala 3! gradle/gradle/pull/18001

Status: ✓

Issue under gradle/gradle/16527

From version: 7.3.0

Unit SAM and overloads

object Reproducer {
  // assertThrows has 3 overloads, 2 with 3 parameters and
  // 1 with only 2 parameters.

  // This overload is taking a subclass of `Throwable`, 
  // and an `Executable` which is a parameterless SAM 
  // returning void
  assertThrows(classOf[IllegalArgumentException],
               () => 3)
	// Compiles

	// This overload is taking a subclass of `Throwable`,
	// an `Executable` which is a parameterless SAM 
	// returning void, and a `String`
	assertThrows(classOf[IllegalArgumentException], 
               () => 3, 
				       "This is a message")
  // Doesn't compile in 3.0

	//This overload is taking a subclass of `Throwable`,
	// an `Executable` which is a parameterless SAM
	// returning void, and a `Supplier` returning `String`
	assertThrows(classOf[IllegalArgumentException],
	             () => 3,
				       () => "This is a message")
  // Doesn't compile in 3.0

	def assertThrows[T <: Throwable](clazz: Class[T], 
	                                 executable: Executable
									                 ):Unit=???
	def assertThrows[T <: Throwable](clazz: Class[T],
									                 executable: Executable,
									                 message: String
									                 ): Unit=???
	def assertThrows[T <: Throwable](clazz: Class[T],
									                 executable: Executable,
									                 supplier: Supplier[String]
									                 ): Unit=???
	
	@FunctionalInterface
	trait Executable {
		@throws[Throwable]
		def execute(): Unit
	}
}
				

Workaround

object Reproducer {
	assertThrows(classOf[IllegalArgumentException],
							 () => 3)

	assertThrows(classOf[IllegalArgumentException], 
							 () => {3; ()}, 
							 "This is a message")

	assertThrows(classOf[IllegalArgumentException],
							 () => {3; ()},
							 () => "This is a message")

}

Status: Not solved ✗

Issue under lampepfl/dotty/issue/13549

It conflicts with existing Scala 3 code making functional code fail.

No static forwarder methods in trait companion

object ObjectTraitPair {
	val Constant: String = "Some Text"
}

// In Scala 2.13 this class bytecode will carry over
// any val and def defined in the object with the same name
// but not in Scala 3.0
trait ObjectTraitPair {
	val method: String = "bye"
}

Bytecode discrepancy

Output of javap ObjectTraitPair.class:

public interface ObjectTraitPair {
	public static void $init$(example.ObjectTraitPair);
	public abstract java.lang.String method();
	public abstract void example$ObjectTraitPair$_setter_$method_$eq(java.lang.String);
}

But it should be:

public interface ObjectTraitPair {
	public static java.lang.String Constant();
	public abstract void example$ObjectTraitPair$_setter_$method_$eq(java.lang.String);
	public abstract java.lang.String method();
	public static void $init$(example.ObjectTraitPair);
}

Status: ✓

Issue under lampepfl/dotty/13572

From Scala version: 3.1.0

Variable handling in super calls

class ClassWithLambda(sup: () => Long)
class ClassWithVar(var msg: String)
		  extends ClassWithLambda(() => 1)
val _ = new ClassWithVar("foo")
// Throws at runtime!
// java.lang.VerifyError: Bad type on operand stack

But this works!

class ClassWithLambda(sup: () => Long)
class ClassWithVar(_msg: String) 
      extends ClassWithLambda(() => 1) {
	var msg: String = _msg
}
val _ = new ClassWithVar("foo")

Status: ✓

Issue under lampepfl/dotty/13630

From Scala version: 3.1.1

Handle Java varargs with parametrized T...

Given this Java class:

public class TypedVarargs<V> {
	public TypedVarargs<V> varArgs(V thing, V... things) {
		return this;
		}
	}

And this Scala one:

val x = new TypedVarargs[Long]()
val y = x.varArgs(1L)
// This throws at runtime: 
// java.lang.ClassCastException: [J cannot be cast to [Ljava.lang.Object;

Workaround

val x = new TypedVarargs[java.lang.Long]()
val y = x.varArgs(1L)

Status: ✓

Issue under lampepfl/dotty/13645

From Scala version: 3.1.1

Type erased for by-name parameters

Given this code:

object ByNameParam {
	def byNameParam(str: => String): Unit = {}
}

Output of javap ByNameParam.class:

public final class ByNameParam {
	public static void byNameParam(scala.Function0);
}

But should be:

public final class ByNameParam {
	public static void byNameParam(scala.Function0<java.lang.String>);
}

Status: ✓ (Prerelease)

Issue under lampepfl/dotty/13638

From Scala version: 3.1.2-RC1

Summary:

<irony>
Easy, huh?
</irony>

What should we improve?

Fading away?

why wasn't it more straightforward?

Java → Scala interoperability improved substantially in 2.12 and 2.13.

Dotty followed kind of a parallel line branching out in 2.11.

Community build

We need more non-fully Scala friendly environment.

To include projects on the fringe of the core of the Scala community.

Mixed Java/Scala projects

We, the Scala community, need to get closer to these projects.

Drop by your local Java/Scala OSS Project!

Further Info:

• Mailing list thread
• Big proof of concept
• In between step
• Blog post that originated this talk

Thanks!