Who?

🔗 Don't Put All Your Code In Internal

text

First published: .
Last modified:   .

I originally titled this rant "Don't Write Internal Packages In Go". I was on full rant mode when I wrote this and made a sweeping statement that wasn't really what I intended to say, and which understandably angered many developers on Reddit, who are already an angry bunch. In that, I sinned in the same way as those I criticised: I generalized something which shouldn't be generalized. I have modified the title and the post to make my point clearer.

Also, please read my follow-up rant.

Go has been developing quite well as a programming language over the past few years. Some of its most annoying shortcomings have been largely solved, and after a lot of frustration in the early days, I can now see clearly how some of its design choices helped me progress as a software engineer.

The language has established a strong footing in the world of free software/open source, and seems to be leading in the rising domain of "cloud native applications."

There's one feature of the language, however, that I often see abused: internal packages. Here's a quick description from the output of go help gopath:

Code in or below a directory named "internal" is importable only by code in the directory tree rooted at the parent of "internal".

Before I explain how this is abused, let's go back to a similar feature that many programming languages share: private functions/methods. The ability to hide functions from consuming code has enough justifiable use-cases that stem from security, healthy abstraction, hiding implementation details which consumers shouldn't depend upon, or other reasons, but the problem with it is that developers often miscalculate which functionality should or shouldn't be private. It is not uncommon for a developer who's using a library to find that useful functions (which would not constitute implementation details) are unusable, because the developer had chosen to make them private, either from an overabundance of caution ("I don't want to maintain usage of this"), or simply making an easy choice.

Now, I should make it clear that this is completely fine. I have no expectations of receiving support, or even of it still being available in later versions of your project. And I also acknowledge that many open source project maintainers (or any software engineer really) are often inundated by support requests from users who depend on behavior simply because it was available to them, even if not part of the "contract".

At this point, you're already good and mad. And you may say that that's the nature of open source software, that I can just fork the project and make whatever changes I wanted to it, and that I can go fork myself. And I agree. But this takes away from one of the main benefits of open source software: the ability of the community to contribute and help the project evolve. So my options are to fork the entire project and maintain this fork (which I do, from time to time); to ask the maintainer to make the change (e.g by opening an issue); or—more appropriately—make the change myself and open a PR (which I also do, from time to time), which may or may not be accepted. Again, this is completely fine.

Internal packages are more extreme. They hide entire implementation details from the user. They also allow you to make your entire project practically private, thus allowing companies to say "hey, look, we're open source!" while completely barring people from using the software outside of the official clients. With Go, this is easy: write the client as your main program (which is not importable due to being an executable), and put everything else under internal. That's it. If your software is popular and ubiquitous enough, you've hit the jackpot. You've created a completely closed software that is still technically open source. No community will be able to fork and maintain your multi-million dollar project. In hiding your implementation choices, you've made an implementation choice for the user.

Again, I should make it clear that I have no expectations here. It's your prerogative, and I will not force you to expose what you're not comfortable with exposing, or truly know shouldn't be exposed. What I'm asking you is to not put your entire codebase under the internal tree.

As an example, let's take HashiCorp's Terraform project. HashiCorp has been building well made software for years, and the popularity of their software is no accident. If you look at the source code for Terraform, however, you will see that virtually the entire software is written as internal packages, only usable by the project itself. Yes, there's a plugin mechanism, but it doesn't provide other avenues of using the product. In the end, the code the company has donated to the community can only be used from the terraform command line interface, and nothing else. You can't import github.com/hashicorp/terraform to integrate Terraform into your project, even as a paid customer. Again, their prerogative, but I believe it's a missed opportunity of making it even more useful.

I sometimes also see internal packages being used extensively in closed source software inside companies. You have projects that depend on each other, but developers put everything under internal because "reasons," and now you have to coordinate between teams.

Listen, I'm not asking you to maintain some function until the end of time for assholes like me. I'd just appreciate it if you give it a little more thought, and refrain from putting everything under internal.

Comments:

After working on a large "enterprise" project where Java developers new to Go built a Go app but did not implement any testing, and I had to come along and build testing for it, I am in VIOLENT AGREEMENT with you. #fwiw

― Mike,

Here is how I got around the internal folder of the fyne code. 

        nil@NIL:~$ cd
        nil@NIL:~$ mkdir workspace_okp
        nil@NIL:~$ cd workspace_okp/
        nil@NIL:~/workspace_okp$ git clone https://github.com/josephbudd/okp
        nil@NIL:~/workspace_okp$ git clone https://github.com/fyne-io/fyne.git
        nil@NIL:~/workspace_okp$ go work init
        nil@NIL:~/workspace_okp$ go work use ./fyne
        nil@NIL:~/workspace_okp$ go work use ./okp

To be more specific. If you look at https://github.com/josephbudd/okp. Step 3 shown below. The workspaces allowed me to copy my own widget into the ./fyne/widget/ folder where the files are allowed access to the internal folder.

So that's how I got access to the internal folder.

Step 3: Add my mousepad widget to fyne.

The source code for my mouse pad widget needs to be copied from a hidden folder in okp to where it belongs in the widget folder in fyne. 

nil@NIL:~/workspace_okp$ cp ./okp/_files/mousepad.go ./fyne/widget/

― Joseph Budd,