Towards A Brighter Tooling Future for Scala

Scala has, for its entire existence, been on the back foot with its tooling support. This was true when the language first arrived in 2004 as little more than a command-line compiler, through 2010 when Typesafe’s (now Lightbend’s) push for commercial adoption took an early focus on improving the Scala IDE for Eclipse, through to today, as the tooling experience is very much improved thanks in part to the work of JetBrains on IntelliJ, but still lags behind the tooling support some other languages enjoy.

People care about language tools, and it says a lot for Scala that its success has come without ever having had the best tooling. But a better development experience in other languages over the years has undoubtedly limited the adoption of Scala.

Now, in 2018, there have been some developments primarily outside of the Scala community which present a new opportunity to change that.

The Language Server Protocol

The language server protocol (LSP) initiative, started by Microsoft, but already seeing wide adoption, is a language-agnostic protocol for allowing tools which consume information about code (“clients”, such as IDEs) to communicate with compilers and other tools which can provide that information (“servers”, or as the protocol describes them, “language smartness providers”). The LSP proposal has had early successes and has collected a lot of momentum behind it. So it’s no surprise that there’s already a lot of enthusiasm within the Scala community around the idea.

During discussions in the last Scala Center Advisory Board meeting, one of the Scala Center sponsors proposed that the Scala Center could support the initiative, so we scheduled a meeting in January to get the ball rolling.

With a many players from both commercial and open-source backgrounds already involved in tool development for Scala, we wanted to make sure that all the developers who had already contributed significantly to Scala’s tooling could attend, but without having so many attendees as to make the meeting unmanageable. So we privately invited about fifty people suggested by the Advisory Board members, more than half of whom attended.

This “closed doors” approach for the initial meeting was a practical compromise, made just for this one-off meeting. There are certainly other people in the community who have contributed to tooling who didn’t get the chance to attend. To those we overlooked: sorry! we certainly want to include everyone who cares deeply about Scala’s tooling, but it wasn’t possible to run a meeting with more than about thirty attendees.

The Initial Meeting

The goals of this first meeting were, nevertheless, modest.

Firstly, with a number of different existing Scala tools which could play a part in Scala’s LSP story, almost nobody in the meeting was familiar with them all, so to get everyone to a common understanding of the ecosystem, the largest part of the meeting was spent hearing different tool authors speak about their tools, and how they might play a part in the jigsaw.

Secondly, we wanted to decide upon what our next steps should be, and how the Scala Center could help the initiative. This was a task of setting up a framework in which decisions about the Scala LSP initiative could be made in future, in a way which the majority of people supported.

During the meeting, we heard presentations on Scalameta and SemanticDB, Metals, Bloop, SBT, Pants, Bazel, Ensime, IntelliJ IDE, Scala IDE, Dotty IDE (Visual Studio Code), and TASTY.

Another important premise for the meeting was that the tooling should be, as much as possible, compiler-agnostic. That is to say, we should not need to use a different protocol to work with Scala 2.11 or 2.12 or Dotty, and nor should users receive a different front-end experience for any Scala-like language. This support for other compilers is important for two reasons, even though the vast majority of current users run Scalac.

Firstly, compilers like Dotty and Rsc want to hit the ground running with their tooling support; waiting another decade for a new tooling ecosystem to be invented is not an option. When the time comes for Scala users to upgrade to Dotty, nobody wants the additional barrier of having to upgrade their build tool, documentation tools or IDE at the same time. They want a seamless transition. That’s much easier to achieve by considering other compilers now than would be later.

The second reason is that there is a huge amount of excitement and enthusiasm around these new compilers, and we want to channel that into the Scala LSP initiative, and take advantage of the commitment from these other developers to pull together towards a common goal.

In the meeting we heard representations from the project leads of the Scalac compiler, Dotty, Rsc and Hydra about any unique features of their compiler implementations that might have any bearing on the LSP initiative.

We didn’t have time during the meeting for much detailed discussion, though the attendees talked about a number of ideas and concerns around what shape the initiative should take and how it should proceed. One distinction that was clear throughout the discussion was the need to separate a Scala implementation of the LSP (the protocol as described my Microsoft), from a more general “build server protocol” which would allow for the exchange of other information specific to Scala-like languages, like dependencies and classpaths, and this is reflected the main outcome of the discussion.

Minutes for the meeting are available in the scalacenter/tooling-working-groups repository

Next Steps

I made a proposal during the meeting that the next steps should include the election of two working groups to take forward ideas for an implementation of LSP for Scala, and a second “Scala Tooling Protocols” group for defining other non-LSP protocols for Scala. These groups should have,

• 6-9 members in each
• some overlap in membership between the two groups
• a mix of experts (more experience than free time) and enthusiasts (more free time than experience)
• diverse representation of a variety of existing projects
• at least one member from the Scala Center, but not overrepresentation

As an open-source project, the dynamics of a disparate collection of developers is often better suited to innovation than shipping working software by a deadline, and a large part of the challenge will be to keep everyone on track, despite the freedom that Open Source grants us. We hope that that leadership can come from the Scala Center.

Election Process

We want to encourage an open process for discussion of tooling for Scala. We want to involve many diverse minds, and we want to be inclusive. In the past, it has often been most expedient to invite a selection of known experts to take part. This has largely led to technical success, but has made it harder for many newcomers to engage with the Scala decision-making process. So to try to improve upon this, and to give the working groups greater legitimacy, we will be holding elections to choose the members of the LSP and STP working groups. This will, in part, be an experiment, but we hope it signals greater openness in how decisions which affect the entire Scala community are made.

If the Scala community had a list of its members, we could invite each member to vote on the makeup of the working groups, but of course, that’s not how the community works. We also do not want to open an election up to anything that could be so trivially compromised as an online poll.

We will therefore take an imperfect but pragmatic approach and open voting only to those members of the Scala community who are willing to devote a small amount of their time to participating in a series of live votes, conducted online, which will take place on Wednesday, 28 February starting at 6pm GMT. Full details of the rules governing this process are described here.

We will announce the composition of the working groups after the voting immediately after voting has finished.

Becoming a Working Group Member

Sitting as a member on either of the working groups is an opportunity to participate and influence an exciting area of development for Scala, and will be a position of some respect and privilege. But bootstrapping these groups in a way that is fair is harder than it might at first seem: the nature of our initial meeting meant that not everyone could be involved, but we do not want to exclude anyone from the opportunity to sit on these working groups.

We feel that it is very important to include a combination of experience and energy in both working groups, so whilst the expertise of their members will be essential to their success, a willingness to participate and contribute are also extremely valuable selection criteria, and anyone lacking experience but having a strong desire to be involved and dedicated to the cause should not hesitate to seek candidacy: that commitment will be very valuable to the working groups.

Candidates to the working group should be nominated by opening a pull request to the scalacenter/tooling-working-groups repository, following the instructions described here, and we strongly encourage anyone with an interest in Scala’s tooling, and a willingness to devote some time to participating in working group discussions to be a candidate.

I will be giving a keynote about the future of Scala Tooling at Scala Sphere in April, a large conference in Kraków dedicated to making Scala developer tools better, now in its third year. I hope to be making some exciting announcements there about our prospects for vastly improved tooling support for Scala! Meanwhile, the working groups will have a challenging task ahead of them, but with a their collective experience, the energy and enthusiasm Scala’s tooling contributors, and the support of the entire community, we can be much more optimistic about that future than ever before.