Go is a particularly good language for CLI's I have found. At least compared to Java/C#/Python.
It's reasonably fast, compiles down to a simple to distribute binary, and the language is forgiving enough that you can do exploratory programming in it. Go-routines make it especially easy to deal with network calls in it as well. For anything that needs absolute performance though look elsewhere, but even then Go might be a good choice for prototyping.
I actually started learning Go with CLI applications. I have found that https://github.com/spf13/cobra tends to be one of the better CLI helpers you can get into but https://github.com/jpillora/opts is one I have been meaning to try following a presentation I saw on it once.
> [Go] is forgiving enough that you can do exploratory programming in it.
I agree with most of your post, but I’m not sure I would describe Go as “forgiving”. In fact, it’s well known for being strict. For example, exploratory programming would be significantly easier if the Go compiler could (optionally) ignore unused variables and unreachable code.
I’ve also found exploratory programming in Go is hindered by needing to frequently cast between different integer types. Maybe I’m doing something wrong, but my Go code is often littered with casts between different integer sizes and signedness. It’s much easier with Python’s arbitrary-precision integers.
package main
import "fmt"
func main() {
var start int = 1
breakOuter:
// for x := start; x < 10; x++ {
for x := 3; x < 10; x++ {
for y := 0; y < 10; y++ {
result := y * 10 + x
// fmt.Printf("Result: %d\n", result)
// if result == 42 {
// fmt.Printf("Breaking")
// break breakOuter
// }
}
}
}
Building:
$ go build
# example
./main.go:3:8: imported and not used: "fmt"
./main.go:8:5: label breakOuter defined and not used
(compilation fails)
$ go build -gcflags=-warnunused
# example
./main.go:8:5: Warning: label breakOuter defined and not used
./main.go:3:8: Warning: imported and not used: "fmt"
./main.go:6:9: Warning: start declared and not used
./main.go:12:13: Warning: result declared and not used
(compilation succeeds)
I had the same opinion you did until I went to a Go meetup and everyone else was a C programmer. Coming from Ruby, Go wasn't expressive or forgiving. But everyone coming from C thought it was wonderful.
Personally, I feel you with respect to casting integer types. I'll code away with int's until something suddenly needs an int64 to use a package and I have to cast everything or refactor everything to int64. I once commented in a thread where people were asked, "In hindsight what feature would you like in Go?" I said that int should just be an alias for int64 (and float == float64), since these were the defaults in the stdlib. I was downvoted into oblivion in a thread on hypotheticals. I understand the historical machine dependent 32/64 difference, but since the stdlib made a choice, the default should line up nicely. That said, I mostly run into this in Project Euler problems, so not in my day to day work.
> Maybe I’m doing something wrong, but my Go code is often littered with casts between different integer sizes and signedness.
It can actually be a feature, and one of the things that brought me to go, as you can define your own integer (or float, or string, etc.) types, thus making them incompatible with each other:
type distance int
type speed int
func distFor(d distance, t time.Time) speed { ... }
...
x = distFor(x, t) // Oops, that's probably a bug!
Not many languages let you do this, especially back then. But yeah, not great for exploratory programming.
This is probably the biggest thing I wish was easier to do in Rust. You can make "newtypes" by wrapping the value in a tuple (that gets compiled away), but it's a fair amount of boilerplate or macros to make the newtype useful. Once you have what you need, it's fine, but it's even less convenient for exploratory programming ;-)
> but it's a fair amount of boilerplate or macros to make the newtype useful.
As mentioned in the docs, you can use the Deref facility to have the newtype implement everything that the original type does. Rust just gives you the choice of doing this vs. wrapping with a custom set of impls.
I have to admit that I felt that way about it too. In fact, I think everybody felt that way about it at first. It was a work-around. The reason it was never changed, though, was that people didn't find anything significantly better. Rust is already a big language. It doesn't necessarily need new constructs.
However, as the OP of this thread, I have to admit that I often feel conflicted about it. Using the Deref trait certainly feels like another hack (and some people really dislike it). Basically Deref is what's used to dereference a reference. It's invoked automatically when you are calling a trait function on a variable. So if you have a variable that's a struct that implements a trait, it will bind to the function on that trait. If your variable contains a reference to the struct, then the compiler is smart enough to see if the struct implements the trait and then automatically uses the Deref train to derefernce the reference and bind to the function.
So if you want to delegate from one type to exactly one other type, you can do it simply by implementing the Deref trait on the first type and having it convert to the second type. It's kind of elegant, but also kind of hacky :-) There are people who feel that Deref should only be used for objects that are actually memory references. Other people feel that it's OK to use it when one type is masquerading as another type (as we are doing when we put a data structure inside a tuple to make it a "new type").
On the plus side, it gives you really fine grained control without adding new constructs to the language. You can make a "new type" that incurs no runtime overhead (in speed or space) and you can choose whether to delegate all of the function calls to the contained type, or to control them explicitly. The former is really, really easy (essentially 5 lines of boilerplate) and the latter is extremely easy to read and reason about. I have to admit that it's hard to justify adding a new construct for something that is not actually broken (except when you first look at it ;-) ).
And, really, to me this is Rust in a nutshell. It's got a lot of really elegant and intelligent decisions going into its implementation, but all of them look incredibly unlikely when you first look at them. The result is that the learning curve is quite high and the road to fluency long. Often newbies (which I probably still qualify for) ask, "Why the heck is it done this way". When you get the answer, it make sense, but it's often not as satisfying as you had initially hoped :-) Still, like others who push past that point, I've got to say that I really enjoy writing Rust code. It's strange.
Ok, let's say that it's more suitable to exploratory programming than other compiled languages? On one hand, yes it's strict about unused variables, but other features (e.g. fast compile times) more than make up for the disadvantages.
What I mean is that while the compiler is strict its fairly easy to throw something together that mostly works. The large standard library really helps there. It's certainly not as easy as in Python though.
Maybe I'm just unreasonable, but it always slightly bothered me when a JS or Python CLI utility has that half second of startup before doing anything, even displaying help. I can't be too annoyed, since in reality it's only a fraction of a second and they're spinning up the entire interpreter.
Single binary is also another one that really shouldn't matter to me, but still does. Especially for small utilities or web services, it's just really nice to know that I can `scp` to my server and just run it if I wanted to, even though, in reality, I always use a Docker container.
CLIs are sorta my ideal use-case for Go. Goroutines are so error-prone to control since you don't have many options for abstraction, so it's relatively difficult to build long-running highly-stable programs...
But CLIs don't usually need that. They can be ctrl-C'd if they go off the rails, and any dangling goroutines just die when the process dies. The simple distribution, fast startup, simple type system, and yolo-concurrency really pay off in pleasantly small and performant tools. And the stdlib is cross platform, quite capable, and very friendly to use. It's almost exactly what you want when you need to go beyond a tiny bash script, to something that might be a up to a couple thousand lines.
I do wish the built-in flags lib wasn't so abhorrent though. Pulling in a replacement lib is step 1 for any CLI.
"Goroutines are so error-prone to control since you don't have many options for abstraction, so it's relatively difficult to build long-running highly-stable programs..."
This comment does not really makes sense, Go #1 usage is for backend services, so it's indeed long running / stable program.
"They can be ctrl-C'd if they go off the rails, and any dangling goroutines just die when the process dies"
There are solution in Go to handle that case, using context and channel but ultimatly it's not a Go problem if you kill an app right away usually there is no way to clean-up everything in a clean way.
His comment makes sense. golang "goroutines" are finicky compared to other ways of doing concurrency (e.g. Futures/Tasks in Scala/C# or Java's upcoming green threads). There are several reasons to this, not limited to the fact that it's trivial to mistakingly ignore return values (especially errors) when executing `go foo()`. Also, there's a lot of boilerplate involved if you want to start several of them and wait until they're completed, or if you want to compose them. The other languages I mentioned, which have superior abstractions to golang, solve the issue in a much better way.
I'd also label it that much of go concurrency control is intrusive, in that you need to write control-flow code into whatever it is you're trying to accomplish, or (typically) give up type safety or understandability. That's unavoidably more error-prone than not writing your own control-flow code, and I spend quite a lot of time discovering and fixing issues around it in libraries, even from some very skilled and careful teams.
Of course there are ways to achieve both in any language, but the massive educational pressure of the official docs and tutorials and examples cannot be ignored, and has profound impacts on what the community ends up building in the language.
---
It's roughly equivalent to manual memory management, IMO. Terabytes have been spilled claiming that C is safe if you're careful enough or use safe patterns, and CVE after CVE provides evidence that nobody is sufficiently careful. It has its benefits, but it also has its downsides.
> Goroutines are so error-prone to control since you don't have many options for abstraction, so it's relatively difficult to build long-running highly-stable programs...
Oh? Building long-running microservices is literally my main use of Go, and in my opinion the language excels at it. What sort of issues are you having?
I'm not OP, but several come to mind: easy to mistakingly ignore return values (such as errors), difficult to compose, difficult to join, boiler plate heavy, no hierarchy/supervisor structuring.
As the typical SysAdmin who likes to automate stuff, I have to agree. I am not experienced enough to talk about language designs. But I can say, that writing some small CLI application and deploying it onto some server is way less work with go. Simply because you can crosscompile the application and generate a standalone binary.
Everytime I deploy some Python3.7 Flask Application on RHEL7 I start to scream.
You were talking about long running applications and broken goroutines: What is the best alternative? I liked to stumble around in Elixir but I feel bad about deploying some application probably nobody at my office will ever be able to 'fix/update'. Python feels too "sluggish". Since I am no Python Pro I have often the feeling, that I am doing something wrong because the language doesn't show any borders.
Alternative: I'd say Java (or similar), since it has the language abilities and sophisticated static analysis tools are readily available. But startup time is still not interactively-fast, so for CLIs it's sorta a no. For long-running processes tho I mostly like it, and stuff like compacting GCs keeps it running healthier much more easily than Go.
Beyond that, dunno - I usually reach for Python since I've written it professionally for a few years and it's pleasantly terse. But it's a fair bit of work to make actually fast and I don't generally think it's worth that effort. Go is much easier there... as long as it's kept simple.
I have some strong hope for Rust, but I think it's fair to label it as "still maturing", though it's already very far of ahead many langs in some areas. And I just don't have much experience with it yet, so have no real conclusions ¯\_(ツ)_/¯
They've been working on it for quite a while, yeah :) I appreciate it and they've made quite a lot of progress... but it's still nowhere near Go, and I mostly doubt it ever will be.
# time ./hellogo
hello world
real 0m0.005s
user 0m0.001s
sys 0m0.005s
golang's gc is non-compacting. I wouldn't be surprised if there are cases where fragmentation becomes too much for a golang service to continue behaving properly. This becomes more likely in long running services.
Interestingly my current CLI project https://github.com/boyter/cs/ does need to be stable because I am putting a TUI mode into it... and yes dealing with the dangling goroutines in it for the TUI mode itself is especially painful.
Generally if you just stick to fan-out-in processing though I find goroutines not too bad.
Check out structopt. It's a declarative layer atop clap. It's a bit polarising, but if you don't mind the "magic" (which you probably don't if you think clap looks nice) it's amazing.
Not sure why this is downvoted. Java, C#, and Python all tend to have slow CLIs. This is very much in-line with my experience. Even if the VMs do start up quickly, people tend to do a lot of expensive initialization, class loading, etc before the program starts. Go’s runtime is minimal by comparison and there is much less work done at initialization (by convention) than in Python, Java, etc.
I work on ML infrastructure (https://github.com/cortexlabs/cortex) and we originally wrote our CLI in Python. Rewriting it in Go has been a major win both in terms of performance and cross-platform support.
Large projects like Hugo and Kubernetes have used Cobra to build their CLI tools, and it's fairly light as well even if you need simpler usage. We use it at my workplace simply for wrapping our microservices and the few commands (serve, migrate, etc)
Don't think cobra and viper should be the default CLI framework. They're useful but overkill for most use case. I personally find that the standard lib has more than enough power to do everything.
I only recently discovered goreleaser and I can't get over how simple it was to add to my CI setup. Distributing binaries was my goal for having a "real" open source project people could use. After a year of avoiding the issue, I discovered goreleaser and got it running in less than an hour.
I wrote a few small cli tools and the first 2 things i always add is urfave cli and goreleaser... even if you only deploy and use local, this is a great way to release getopt compilant multiple platform builds with or without subcommands.
I love using Go for CLIs. Previously, I developed CLIs with JavaScript (NodeGH, something with the same goal as GitHub's CLI), and "kind of" with PHP as well (for internal tasks on a server). Nothing compares with writing one with Go, though.
It's currently in Beta but the first 6 chapters are finished and available. As someone learning Go I found it a nice complement to reading The Go Programming Language.
I've been building CLIs with Go as part of a 52 projects in 52 weeks challenge[1] and I'm loving it so far. I released a batch renaming tool [2] just a few days ago.
Whatever you do, do not use the flag library in a package that might ever, EVER be imported. Google did this in the horribly written glog port to go which until recently was used everywhere in Kubernetes. The only way to determine the value of the "v" flag they define globally for your entire executable is to call the V(n) function with n incrementing until it returns false.
pflag, which is used by cobra, is a much nicer library.
I wrote my own flags package called flaggy and think its the easiest to use and makes the most sense! Up to 600 stars om github now.
https://github.com/integrii/flaggy
Terminal is simultaneously powerful and painful tool. I know a guy that refuses to use anything but CLI and suffers a lot. But most basic apps can be written it in like those BIOS menus from a 2005 dell computer.
You may be unfamiliar with the history of terminals, in which case this explanation is for you.
What you might think of as the terminal is really a terminal emulator. A terminal was a device with a screen and keyboard (early terminals like the Teletype were printers with keyboards, or typewriters). Most could only display text. Up until the late 1970s or so, computers were invariably large and most did not come with their own built-in, memory-mapped display hardware. To access them, you had to use one of these terminals which connected usually via a serial cable.
Because Unix was designed to work with these terminals as the user's means of interaction, terminal emulators which spoke the same protocol as the actual terminals became common. When Unix got graphical displays, terminal emulators were based on them so that old, text-based Unix programs could continue to be used in the graphical environment.
Some graphical protocols for terminals emerged. xterm supports three such protocols: Tektronix 4014, DEC Sixel, and DEC ReGIS. Sixel can be used on xterm to transmit color images. However, Sixel support is usually not compiled in by default.
Perhaps the most sophisticated graphical protocol for terminals to emerge into common use was X itself. Yes, vendors used to sell X terminals that were basically bare-bones computers without any local storage, just a network connection that allowed it to present a windowed display from a remote machine.
It depends on the console/terminal. iTerm has a native ability to do this, the built-in Linux console has some programs like FIM http://www.nongnu.org/fbi-improved/.
I am imagining like a gallery app such as Instagram purely in terminal lol never mind the consequences of the user base being primarily sys admins and software engineers. I could imagine browsing a rather weird Instagram clone.
The problem with Python is that it’s really hard to distribute a Python app to users. Nothing beats Go’s ability to compile into a self-contained binary.
Python is also slow and has poor support for parallelism.
Finally, “Python is simple” only really applies to its syntax. Overall, Go is a significantly simpler language than Python.
"Slow" isn't a typically big deal for CLI apps, but Python culture tends to do a lot of work on initialization, and I've consequently seen a lot of Python CLIs that take nearly 10 seconds to print --help. Because the way most apps and libraries are written, you end up loading and initializing every single library that any part of your CLI uses, even if the subcommand in question doesn't use those libraries (e.g., printing --help). This is solvable via lazy imports, but few go through the hassle of doing this, especially since it makes the code ugly and non-idiomatic.
I do like Python, but I think Perl is actually designed (and actually is) a super-charged bash. Perl was created with the purpose of simplifying the combination of bash + awk + sed + grep, a goal it has achieved quite well.
Python has better OO and has won more mindshare than Perl, but I don't think it was ever meant (or accomplishes the goal) of being an "overcharged" bash.
Is Go a good option to create some background tasks. For eg. monitor which Spotify songs are playing and and store that in a json file, etc. Or is Python more suitable for this?
Python has more libraries to write automation tasks than Go. It really depends on you situation, if I have a lot of time I'll do it in Go because everybody does it in Python :P
I had a similar idea a couple of months ago. Make a CLI in golang that allows me to easily open all my favorite timewaster sites at the start of the work day. But it ended up in the "Started, but never looked at again" pile. Maybe I'll have another look at it on the weekend.
One method I've used which puts some additional pressure on me to work on side projects is to write them out in the open in a public GitHub repo. Even though there's probably nobody following your development, the idea that anyone could be following along can provide some motivation to keep going.
Is your goal to learn go (or other language), or to accomplish a specific task (both valid and wonderful goals!)? The reason I ask because if it's the second, I've become dramatically more productive by really learning shell. Just as an example:
That took maybe a minute for me to write, vs. like 20 or so minutes for Go. I personally can get stuck in a such a paralysis of doing things the "right" way in a "real" language. So, it's nice to be able to bang out a prototype super quickly and then iterate from there if I want.
edit: getting them all, because who can resist some fun code golf:
I personally love building CLI tools in Node, since I and my co-workers all have it installed. Nice synchronous STD lib for file manipulation and async/await makes for compact async code.
If I worked in a Go shop I'd probably use Go, though.
"Start up a whole async event loop" is really just "call one of the event loop APIs in the kernel". It's not like "starting an event loop" is intrinsically costing a millisecond and half a gig of RAM or something.
What a larger system builds around the event loop may be heavyweight, but I think that would generally be less about "using Node" or "using Go" and more about picking up some heavyweight framework within the event-looping language. The event loop itself is generally going to be too light-weight to worry about compared to the things it is waiting on to worry about, until you get to a scale that you're unlikely to reach on a CLI.
Most of the tools we have use less than 100mb of ram - and a good number under 50mb. Point is, Node is pretty light for how fast you can build tools with it.
All my co-workers have the same version of Node, since we code in Node, so that argument is mute. My point was it depends on your environment! :)
Node is nice, but maybe a bit verbose and the CLI utilities are a bit low-level. The main downside to Node is that you need an external runtime to run your applications.
But I guess the counter-downside to Go is that you need to compile different versions for different platforms; the binaries are not portable.
Maybe not, but Go has excellent support for cross-compilation. You can still support users on multiple platforms while only developing and building on one.
Here are some arbitrary reasons that Go is worth a look:
It's easy to learn, tractable, consistent, performant, and safe.
It's great for writing services. The type system is predictable and catches a lot of problems ahead of time.
It's opinionated about basic stuff which helps avoid clutter (no unused variables or imports allowed, etc).
It fills a lot of the same niche as Python, Java, Ruby, etc, but with some distinct advantages when compared to each of them (and some drawbacks depending on your preferences).
Binaries are standalone, easy to compile (cross compilation), and easy to distribute.
Seems overcooked in some places and undercooked in others. I of course prefer my own CLI template but so it goes: https://github.com/carlmjohnson/go-cli
While following along locally I got an unhelpful error the first time I ran the program. Ultimately the bug was that I was using a lowercase 'o' instead of a zero '0' in the url builder. The API returned a 404, but the code ignored that and tried to marshal the html error response to the struct and failed.
For your guide, it would be helpful to include a status code check on the response and returning a helpful error message around that.
You don’t need separate client and models packages. You should have two packages: main (which has only one line) and everything else (call it package xkcd).
Your main continues even after it finds an error in fetching the comic. This is because you can’t return err in main and you didn’t use panic/log.Fatal. You also don’t exit with a non-zero code on error.
The solution to all of this is the one line main function. Main should look something like os.Exit(xkcd.Run(os.Args[1:])). I have a helper package at github.com/carlmjohnson/exitcode so you can return an error with an associated exit code, but you can also be simple and just return an int.
Separate flag gathering/processing from execution. You’re not using global flags, which is a good step, but you should go further and make an appEnv struct that consolidates what you’re taking in from the flags. See here: https://play.golang.org/p/-gs5nqXBSuB
Your client package basically doesn’t need to exist. What you want are some simple convenience wrappers around the http package and a URL builder for XKCD. Make something generic for HTTP and you can copypasta it into your future projects. Basically, it just needs to be httphelper.GetJSON(cl ∗http.Client, url string, data interface{}) error. Saving to disk is a separate idea that you’re conflating with downloading. Make something like jsonhelper.SaveToDisk(path string, data interface{}) error. The timeout stuff you’re doing is overblown. Either just use context.WithTimeout or put a ∗http.Client in your appEnv and have ParseArgs set the default timeout on that.
The stuff with the base URL is unnecessary. Obviously you know the XKCD base URL is a constant and will never change. But don’t you need to set it in an XKCDClient struct for testing purposes? The answer is no. If you take in a ∗http.Client, that can set a different http.RoundTripper for testing purposes and the test RoundTripper can read from memory or do whatever you want. (You can see this principle at work in Google’s Go http libraries. Once you realize how powerful ∗http.Client is, it makes a lot of the hoops other libraries jump through see like a waste of time. It can do all your auth stuff, caching stuff, everything. It’s great.)
The model package should just be a file in your xkcd package. I find the names Comic vs. ComicResponse confusing. Do you need Comic at all? Maybe just add some nice helper methods onto ComicResponse. Don’t do cr.FormattedDate() string. Do cr.Date() time.Time and let the output layers handler formatting, not the model layer. The c.JSON() string method doesn’t need to exist. With Go, you can run into this problem of trying to make things more convenient by adding helpers but you end up with methods that just run two commands and don’t actually make things more convenient on net. Is this really a model level concern or should it just be in the xkcd.Run() function?
Anyway, not to be overly negative. For such a small app, none of this really matters. I’ve been making a lot of Go CLIs for a long time[1] and my experience is that the most important thing is to separate flag stuff from execution, and everything else is not a big deal to let evolve over time. The main challenge is avoiding create abstractions that don’t actually pay for themselves in setup time vs. time saved in extension.
I really appreciate you effort in pointing out the correct way to do things.
In my defence I would like to point that this post was to help beginners get hands on experience writing Go code, adding design patterns or organising the code base to make things "correct" will only confuse a person who has just started to learn Go.
And to be frank, I too don't have much practical knowledge about it. If you don't mind can you point me in the right direction?
It's reasonably fast, compiles down to a simple to distribute binary, and the language is forgiving enough that you can do exploratory programming in it. Go-routines make it especially easy to deal with network calls in it as well. For anything that needs absolute performance though look elsewhere, but even then Go might be a good choice for prototyping.
I actually started learning Go with CLI applications. I have found that https://github.com/spf13/cobra tends to be one of the better CLI helpers you can get into but https://github.com/jpillora/opts is one I have been meaning to try following a presentation I saw on it once.