Mastering Go 19: Downloading Multiple URLs Using Goroutines and Channels in Go

👶 “Imagine you have a list of websites and want to open them all at once, not one by one. Go lets us do that easily with goroutines and channels!”

If you were using PHP or JavaScript, you might loop through URLs like:

foreach ($urls as $url) {
    download($url);
}

But this does it one at a time, and can be slow.

What if we want to download everything at once?

That’s where Go shines. ✨

Go makes concurrency super easy.

We’ll:

1. Create a list of URLs

2. Start a goroutine for each one

3. Send back the result via a channel

Here’s a complete Go program:

package main
import (
  "fmt"
  "io/ioutil"
  "net/http"
  "time"
)
func main() {
  start := time.Now()
  // List of URLs
  urls := []string{
    "https://golang.org",
    "https://gobyexample.com",
    "https://example.com",
  }
  // Make a channel to receive the results
  results := make(chan string)
  // Launch a goroutine for each URL
  for _, url := range urls {
    go fetch(url, results)
  }
  // Receive all results
  for i := 0; i < len(urls); i++ {
    fmt.Println(<-results)
  }
  fmt.Printf("Done in %s\n", time.Since(start))
}
// fetch downloads the URL and sends a message to the channel
func fetch(url string, ch chan<- string) {
  start := time.Now()
  resp, err := http.Get(url)
  if err != nil {
    ch <- fmt.Sprintf("Error fetching %s: %v", url, err)
  return
  }
  defer resp.Body.Close()
  _, _ = ioutil.ReadAll(resp.Body) // We don’t need the content
  secs := time.Since(start).Seconds()
  ch <- fmt.Sprintf("%s took %.2fs", url, secs)
}

• go fetch(...): Starts each download in a new goroutine

• ch <- result: Sends message back to channel

• fmt.Println(<-results): Waits for and prints result

• It all runs in parallel, super fast!

ConceptMeaningGoroutineA mini thread — like multitasking for your programChannelA pipe to send messages between goroutinesgo keywordLaunches a function in the background<- operatorReads from or writes to a channel

• In many languages, writing parallel code is hard 😣

• But in Go, it’s built in and easy 😎

• You can download 100s of URLs without stress

Change the URLs in the list — try downloading:

• Your favorite blogs

• API responses

• Image files (just for test)

Maybe even log the status code like resp.StatusCode.

If you want to avoid blocking, you can use:

results := make(chan string, len(urls))

This is like saying:

“Hey channel, you can hold this many messages before I come back.”

• Use goroutines for parallel downloads

• Use channels to collect results

• Simple, readable, and fast ⚡