--- title: Creating This Website description: How I built my website using Go keywords: ssg, static site generator, go, hugo, templating, html, markdown, goldmark created: 2024-01-15 updated: 2024-01-16 --- Over the years my personal website has evolved and undergone _a lot_ of changes. Here's a none comprehensive list of where stigsen.xyz has been hosted: - GitHub Pages (3-4 times) - Codeberg Pages (2 times) - BearBlog (once) - Porkbun (Static Site) (once) - VPS (3 times) I've gone through these changes many many times, because I've been trying to find an optimal way to set up my website, in a way that I like. There are some benefits to all of the options above, but I've finally landed on "self"-hosting with a VPS _(again)_. HTML is really nice for simple blogs. Links, headings, lists and more look completely fine as raw HTML, but code, does not. During my time writing [Developing an HTMX Alternative](../developing-an-htmx-alternative/), while enjoyable, the HTML did not look nice. It looked something like this: ```html
<h1>Hello World!</h1>
<button data-hxm-req="get" data-hxm-url="/content">
	Get content
</button>

<fieldset>
	<legend><b>Form</b></legend>

	<p>Type your information:</p>
	<form data-hxm-req="post" data-hxm-target="p" data-hxm-swap="beforeend">
		<label>Name <input type="text" name="name" /></label>
		<input type="submit" />
	</form>
</fieldset>
``` If you're wondering why, this is because you display the `<` character in HTML by using `<`. Since it's usually used for an HTML tag of some kind, it has to be escaped to show the actual character. After a while I found out that by removing the `` element, I could add the following rule to CSS: ```css pre:first-line { line-height: 0; } ``` This made it possible to write the code above like so: ```html
<h1>Hello World!</h1>
<button data-hxm-req="get" data-hxm-url="/content">
	Get content
</button>

<fieldset>
	<legend><b>Form</b></legend>

	<p>Type your information:</p>
	<form data-hxm-req="post" data-hxm-target="p" data-hxm-swap="beforeend">
		<label>Name <input type="text" name="name" /></label>
		<input type="submit" />
	</form>
</fieldset>
``` Which, while still not great, looked better. I do like writing Markdown though, it's a very simple and clean format, even though it has its shortcomings. Having created a static site generator in the past, I know that it could quickly get complicated to create. I do not like big systems such as Hugo, Jekyll and so on, so I've always opted-in to making my own solution, this included my own Markdown regex parser, which was hell when it came to nested lists. Anyways, as I'm writing this, it has only been a few hours, since I wrote a new static site generator, that I'm really happy with. My static site generator, like the previous ones I have made, is written in Go. But since the last time, I've created several projects where I've learnt how to use Go HTML templates properly. This time I also decided _not_ to write my own Markdown solution, but instead use [goldmark](https://github.com/yuin/goldmark), the same Markdown to html generator used by Hugo. For HTTP routing I use [Chi](https://github.com/go-chi/chi), a module I've used a lot and come to love. It's simple, small, easy to extend and only depends on the standard Go library. It also has some very convienient middleware like basicauth, ratelimit and more. Creating your own middleware is also really simple. So anyways, let's jump into some of the design decisions for this webserver. ## Design I knew that I wanted to not only use Markdown for writing blog posts and so on, but I'd also like to enable other projects to work. As can be seen with my [solaris](/projects/solaris/) project. I'd like to make use of Go templating, in an efficient way productivity-wise, where it wouldn't be me writing the same HTML header in each Markdown file and so on. I wanted it to be a "change once, work everywhere" sort of deal. It should be easy to serve static assets such as JavaScript, CSS, images and so on. In my previous projects I'd do something like the following: ```go router := chi.NewRouter() router.Get("/assets/*", func (res http.ResponseWriter, req *http.Request) { path := chi.URLParam(req, "*") if asset, ok := assetCache[path]; ok { res.Header().Add("Content-Type", asset.mimetype) res.Write(asset.data) return } data, err := os.ReadFile(filepath.Join("static", "assets", filepath.FromSlash(path))) if err != nil { log.Panic(err) } // very complicated way to detect mimetype here assetCache[path] = Asset{path, data, mimetype} res.Header().Add("Content-Type", mimetype) res.Write(data) }) ``` Since then I've gotten smarter and found out that you can just write: ```go router.Handle("/assets/*", http.StripPrefix("/assets/", http.FileServer(http.Dir("static/assets")))) ``` It wouldn't be a solution for younger me, if it wasn't an overly complicated caching system 😎 ```go type Page struct { filename string data []byte title string description string // so on... } type Asset struct { filename string data []byte mimetype string } pageCache := map[string]Page{} assetCache := map[string]Asset{} ``` While I do like caching content, to reduce latency, minimize disk reads and have content readily available, I did not want to do it the same way this time. The code above was accompanied by many functions, to load the asset, figure out the mimetype and for pages specifically a lot of it was extracting the data from the page it self. I later changed this to JSON, so that I could store the description, keywords and so on in another file, but I was adding complexity to an already complex solution. Now I'd have folders with Markdown files, HTML and JSON. Add a field to the `Page` struct and I'd have to modify all JSON files, because I was handling it _very_ poorly. For the Markdown, the way my previous solution worked, was that I'd loop through all directories, collect the `.md` files, extract some data from them, convert them to HTML, save those HTML files. ... anyways, with all of the pain points above (several I also haven't mentioned), let's jump into the development! ## Development ### Initial I have a template for Go web projects using Chi, which looks something like the following: ```go package main import ( "html/template" "net/http" "github.com/caddyserver/certmagic" "github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5/middleware" ) // Global variable for HTML templates templates *template.Template func main() { production := flag.Bool("production", false, "production mode") flag.Parse() go func() { for { templates = template.Must(template.ParseGlob("static/html/pages/*.html")) templates = template.Must(templates.ParseGlob("static/html/components/*.html")) // Reload templates every 12 hours in production mode // Reload them every two seconds in non-production mode, for quick development if *production { time.Sleep(12 * time.Hour) } else { time.Sleep(2 * time.Second) } } }() router := chi.NewRouter() router.Use(middleware.Logger) router.Use(middleware.Recoverer) router.Handle("/assets/*", http.StripPrefix("/assets/", http.FileServer(http.Dir("static/assets")))) router.Get("/", func (res http.ResponseWriter, req *http.Request) { templates.ExecuteTemplate(res, "index.html", nil) }) // Launch with SSL / HTTPS in production mode else launch on localhost if *production { domain := "stigsen.xyz" certmagic.DefaultACME.Agreed = true certmagic.DefaultACME.Email = "myemail@lmao.com" certmagic.HTTPS([]string{domain, "www." + domain}, router) } else { domain := "localhost:1337" http.ListenAndServe(domain, router) } } ``` Okay, there's _a lot_ going on here, a quick summary would be: - We reload templates often in non-production mode, less often in production mode - The Chi logging and recoverer middleware is being used for all requests - When someone requests `/assets/*` they receive assets from `static/assets/*` - We serve `index.html` template / page on the `/` route - If we're in production mode, we use `certmagic` to get TLS / SSL / HTTPS for our domain This is what it looked like at the very start. ### Markdown->HTML and Data Extraction Let's start converting Markdown to HTML. ```go import ( // ... "github.com/yuin/goldmark" "github.com/yuin/goldmark/parser" ) var markdown goldmark.Markdown // ... func main() { // ... markdown = initGoldmark() // ... } func initGoldmark() goldmark.Markdown { return goldmark.New(goldmark.WithParserOptions(parser.WithAutoHeadingID())) } // load Markdown file and convert to HTML func load(path string) (map[string]any, error) { data, err := os.ReadFile(path) if err != nil { return nil, err } text := string(data) content := strings.SplitN(text, "\n", 2) // extract the first line and everything after title := strings.TrimPrefix(content[0], "# ") // get the title after the # body := strings.TrimSpace(content[1]) // markdown body content // convert to html bytes := bytes.Buffer{} if err := markdown.Convert(data, &bytes); err != nil { log.Fatal(err) return nil } output := map[string]any{ "title": title, "body": template.HTML(bytes.String()) } return output, nil } ``` Alright, now we initialize the goldmark parser and generator with a single option, which is to generate auto IDs for the titles. So if the Markdown content contains `## Hello`, this will become `

Hello

`, which allows you to enter `stigsen.xyz/somepage#hello`, which will scroll down to that heading. We create a function to load the Markdown file from a path, extract the title and body from it, convert it to HTML and return the map containing the data. ### HTML Templates Now we can make use of these functions, to get started serving Markdown files as HTML, but first we should add our templates. ``` static/ html/ pages/ index.html components/ head.html pages/ index.md ``` This is `static/html/components/head.html`: ```html {{ define "head" }} {{ .title }} {{ end }} ``` This is `static/html/pages/index.html`: ```html {{ template "head" . }}

{{ .title }}

{{ .body }} ``` This let's us write the following code: ```go func main() { // ... router.Get("/", func(res http.ResponseWriter, req *http.Request) { // load the Markdown file data as HTML data, err := load(filepath.Join("pages", "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "index.html", data) }) } ``` Now when we go to `/` it calls `load()` with `"pages/index.md"`, which converts our Markdown to HTML, then it splits our data into a `map[string]any` which may be something like `{"title": "My Website", "body": "Welcome world!"}`. We then pass this map to our `index.html` template. ### Blog Posts and Directory Structure This is a good start, now we have the following structure: ``` static/ html/ components/ head.html pages/ index.html pages/ index.md ``` But now I want to add the ability to write blogs as well. I'm thinking this should just be a folder within `pages` like so: ``` static/ html/ components/ head.html pages/ blog.html index.html pages/ index.md blog/ my-programming-journey.md ``` Let's add some code for it: ```go func main() { // ... router.Get("/blog/{title}", func(res http.ResponseWriter, req *http.Request) { data, err := load(filepath.Join("pages", "blog", chi.URLParam(req, "title") + ".md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "blog.html", data) }) } ``` I don't want the user to go to `stigsen.xyz/blog/my-post.md`, I'd prefer if it was `stigsen.xyz/blog/my-post`, so I'll simply grab the title parameter from the request, then append `.md` to it. If the file hasn't been found, we return 404 else we execute the `blog.html` template page with the data. At this point I decided to use an online HTML to Markdown translator, to convert `my-programming-journey.html` to `my-programming-journey.md`, to test if it works. Now I ran into something that I've run into before... where do I store blog specific assets? The document contains an image `image.png`, so where do I store this image? Last time I did this: ``` static/ assets/ images/ blog/ my-programming-journey/ image.png pages/ blog/ my-programming-journey.md ``` I quickly found out that it's not a good way to organize the files. After a moment of thinking I decided to change it to: ``` pages/ blog/ my-programming-journey/ index.md image.png ``` This groups the assets nicely together into the appropiate folders, but we'll still use `static/` for "global" content, such as stylesheets and general purpose scripts. To make this change though, I need to modify my code a bit, since we are now looking for `title` folder, not just `title.md` ```go func main() { // ... // Blog Page router.Get("/blog/{title}", func(res http.ResponseWriter, req *http.Request) { data, err := load(filepath.Join("pages", "blog", chi.URLParam(req, "title"), "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "blog.html", data) }) // Blog Asset router.Get("/blog/{title}/{filename}", func(res http.ResponseWriter, req *http.Request) { title := chi.URLParam(req, "title") filename := chi.URLParam(req, "filename") http.ServeFile(res, req, filepath.Join("pages", "blog", title, filename)) }) // ... } ``` There we go, now we can serve assets from the same directory as the blog post. This allows the Markdown to contain `![image](./image.png)` instead of `![image](/assets/images/my-programming-journey/image.png)`. ### Projects I currently have some projects like [solaris](/projects/solaris/), that I'd like to serve. Ideally I'd like to keep them as is, since they aren't focused on Markdown content. This is basically what it looks like: ``` projects/ solaris/ index.html assets/ css/ js/ ``` Since it's a self-contained project, with uncommon scripts and stylesheets not used anywhere else on the page, it doesn't seem productive nor smart to change the entire structure of it. Luckily this was very simple to add support for: ```go // ... func main() { // ... router.Handle("/projects/*", http.StripPrefix("/projects/", http.FileServer(http.Dir("projects")))) // ... } ``` That is it. Now it's possible to navigate to `/projects/solaris/` and it serves the files and assets there correctly. ### Caching Alright, all of the stuff above is great and all, but currently we load Markdown files, extract data, convert to HTML and perform template operations on every single request... this does not seem optimal. Having told you about my caching fiasco earlier, I carefully considered how to go about creating a caching system. Having created my own middleware for the routing library in the past, some ideas came to mind. What if I could just cache an entire response? The last step before responding is template executing, so if I could cache the fully generated HTML from the template, after Markdown processing, then the computer should be able to avoid a lot of redundant work. This is what I came up with: ```go import ( "bytes" "log" "net/http" "sync" ) var cache = new(sync.Map) type cacheWriter struct { http.ResponseWriter buf *bytes.Buffer statusCode int } func (cw *cacheWriter) Write(b []byte) (int, error) { return cw.buf.Write(b) } func (cw *cacheWriter) WriteHeader(statusCode int) { cw.statusCode = statusCode cw.ResponseWriter.WriteHeader(statusCode) } func Cache(next http.Handler) http.Handler { return http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) { if data, ok := cache.Load(req.URL.Path); ok { res.Write(data.([]byte)) return } cacheWriter := &cacheWriter{ResponseWriter: res, buf: new(bytes.Buffer)} next.ServeHTTP(cacheWriter, req) res.Write(cacheWriter.buf.Bytes()) log.Println("Caching response for", req.URL.Path) cache.Store(r.URL.Path, cacheWriter.buf.Bytes()) }) } func ClearCache() { cache.Range(func(key interface{}, value interface{}) bool { cache.Delete(key) return true }) } ``` By embedding the `http.ResponseWriter` interface in a struct, we can now pass this struct along in the chain, so that when at some point the routes call `.Write()`, it's not actually written to the end user, but to our own `bytes.Buffer`. When the functions return at some point and we get back to our `Cache()` middleware, we can now store the data written to the buffer and forward this data to the user. Since the HTTP modules separates requests into their own goroutines, we need our cache to consist of a `sync.Map` instead of a regular `map`. This is because if something has not been cached, we may be writing to the map, while content is being changed, creating race conditions. We also can't clear the cache by creating a new `sync.Map` due to the same reason. But it is safe to loop over the map and delete the content, since `sync.Map` handles concurrent access automatically. The keys for this map is just the entire URL path. So if you go to `/blog/my-programming-journey/` this ends up caching `/blog/my-programming-journey/index.md` after it has gone through all of the HTML and template generation, so we save a significant amount of processing. This cache is simply stored in memory, which for a small blog site, shouldn't be a problem at all. Since it's middleware we can also choose not to cache specific content such as `/assets/css/style.css` or something similar. Oh, and let's also split the routes into nice small functions. ```go func main() { // ... router.Group(func(router chi.Router) { if *production { router.Use(Cache) // cache middleware } router.Get("/", IndexPage) router.Get("/blog/{title}", BlogPage) }) router.Get("/blog/{title}/{filename}", BlogAsset) router.Handle("/assets/*", http.StripPrefix("/assets/", http.FileServer(http.Dir("static/assets")))) // ... } func IndexPage() { /* ... */ } func BlogPage() { /* ... */ } func BlogAsset() { /* ... */ } ``` I made the caching middleware, only run if we're in production mode (which is enabled with the `-production` flag). This allows for quick development, where we don't show the same output, since we want to use the latest templates and markdown content in development mode. We'll also clear the cache when the templates are reloaded in production mode: ```go func main() { // ... go func() { for { templates = template.Must(template.ParseGlob("static/html/pages/*.html")) templates = template.Must(templates.ParseGlob("static/html/components/*.html")) // Reload templates and clear cache every 12 hours in production mode // Reload templates every two seconds in non-production mode, for quick development if *production { time.Sleep(12 * time.Hour) ClearCache() } else { time.Sleep(2 * time.Second) } } }() } ``` While storing in memory probably won't be a problem, it very easily can be if we don't take something important into consideration. Let's say you go to `/blog/foo`, it doesn't exist, but we still cache the result. So with a bit of malicious intent, it's very easy to make the server cache a whole lot of useless responses. A simple way around this is not caching results that have a status code other than `200`. ```go func Cache(next http.Handler) http.Handler { return http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) { if data, ok := cache.Load(req.URL.Path); ok { /* ... */ } cacheWriter := &cacheWriter{ ResponseWriter: res, buf: new(bytes.Buffer), statusCode: http.StatusOK, // default: 200 } next.ServeHTTP(cacheWriter, req) res.Write(cacheWriter.buf.Bytes()) if cacheWriter.statusCode != http.StatusOK { return } // ... }) } ``` ### Simplification One thing you may have noticed is that we currently serve `/` and `/blog/` seperately, but it's actually not necessary. Both are rooted in the `pages/` directory and we load the content from `index.md`. So it goes from this: ```go func main() { // ... router.Get("/", IndexPage) router.Get("/blog/{title}", BlogPage) // ... } func IndexPage() { data, err := load(filepath.Join("pages", "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "index.html", data) } func BlogPage() { data, err := load(filepath.Join("pages", "blog", chi.URLParam(req, "title"), "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "blog.html", data) } ``` To this: ```go func main() { // ... router.Get("/*", Page) // ... } func Page() { data, err := load(filepath.Join("pages", chi.URLParam(req, "*"), "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "page.html", data) } ``` Let's also remove `index.html` and `blog.html`, let's just combine it into a single `page.html`. This introduces a new problem. Because if you may have noticed, the very first line of this page, the title, looks like: **[<](/) Creating This Website**, with a link back to the homepage. I do not want the `<` link to be shown on the index page... it wouldn't make any sense. So we'll just do a quick comparison, to check if we're requesting the homepage. ```go func Page(res http.ResponseWriter, req *http.Request) { // ... data["isIndex"] = chi.URLParam(req, "*") == "" templates.ExecuteTemplate(res, page, data) } ``` Then we modify our `page.html` template a tiny bit: ```html {{ template "head" . }} {{ if .isIndex }}

{{ .title }}

{{ else }}

< {{ .title }}

{{ end }} {{ .body }} ``` ## Configuration (title, description, keywords, etc.) While this works nicely, how would I add something like a description and keywords to the template component `head.html`? I can start by modifying the template: ```html {{ define "head" }} {{ .title }} {{ end }} ``` What I do right now is I extract the title from the first line in the Markdown document... so should I extract the description from the second line? What if I want to add something like `{{ .created }}` and `{{ .updated }}`? It becomes more and more complicated to extract from the markdown. Initially my solution was an `info.cfg` file, containing something like: ```cfg title = Developing an HTMX Alternative description = How I developed a smaller alternative to htmx keywords = htmx, javascript, xmlhttprequest ``` Then I'd load the `info.cfg` in the specific page directory, split on `=` and collect the keys and values. But, I found a much simpler way to do this, which is just to add the goldmark module [meta](github.com/yuin/goldmark-meta), which allows you to have YAML metadata in Markdown file header like so: ```markdown --- title: Developing an HTMX Alternative description: How I developed a smaller alternative to htmx keywords: htmx, javascript, xmlhttprequest --- Content starts here ``` Our `initGoldmark()` now looks like the following: ```go import ( // ... meta "github.com/yuin/goldmark-meta" // ... ) // ... func initGoldmark() goldmark.Markdown { return goldmark.New( goldmark.WithExtensions( meta.Meta, // collect yaml metadata // ... ), // ... ) } ``` Now I can remove the Markdown document extraction logic and change the `load()` function from: ```go func load(path string) (map[string]any, error) { // ... data, err := os.ReadFile(path) if err != nil { return nil, err } text := string(data) content := strings.SplitN(text, "\n", 2) title := strings.TrimPrefix(content[0], "# ") body := strings.TrimSpace(content[1]) output := map[string]any{ "title": title, // ... } // ... } ``` To: ```go func load(path string) (map[string]any, error) { data, err := os.ReadFile(path) if err != nil { return nil, err } bytes := bytes.Buffer{} context := parser.NewContext() // context to store metadata in if err := markdown.Convert(data, &bytes, parser.WithContext(context)); err != nil { log.Fatal(err) return nil, err } metadata := meta.Get(context) metadata["body"] = template.HTML(output) // set the "body" key equal to the HTML content return metadata, nil } ``` Now my `Page()` function looks like: ```go func Page(res http.ResponseWriter, req *http.Request) { data, err := load(filepath.Join("pages", chi.URLParam(req, "*"), "index.md")) if err != nil { http.NotFound(res, req) return } templates.ExecuteTemplate(res, "page.html", data) } ``` I don't even have to check if we're on the index page anymore, as my `pages/index.md` now contains the following at the top: ```markdown --- title: Stigsen XYZ isIndex: true --- ``` ## Syntax Highlighting While code within regular `
` works fine, there's no syntax highlighting. I'd like my site to work
without JavaScript, so making use of Prisma.js or highlight.js is out of the question.
Lucky for us, a syntax highlighting extension has already been created for goldmark, so we can add syntax highlighting
on the server side. `initGoldmark()` now looks like this:
```go
import (
	// ...
	chromahtml "github.com/alecthomas/chroma/v2/formatters/html"
	highlighting "github.com/yuin/goldmark-highlighting/v2"
	// ...
)

// ...

func initGoldmark() goldmark.Markdown {
	return goldmark.New(
		goldmark.WithExtensions(
			// ...
			// syntax highlighting extension
			highlighting.NewHighlighting(
				highlighting.WithStyle("dracula"),
				highlighting.WithFormatOptions(
					chromahtml.TabWidth(4),
				),
			),
		),
		// ...
	)
}
```

This code is what adds the syntax highlighting that you see above!

## Hyperlink Page Navigation
As you see on the title right above, there's a little `#` next to it, which allows you to
link to specific sections on the page. Since we already added auto heading ID to the goldmark parser,
it's already possible to navigate to `stigsen.xyz/creating-this-website#hyperlink-page-navigation`. But
I'd like there to be a little link element next to it. I could add these manually in Markdown no problem, _but_
there's also a simple goldmark extension which makes this easy to do:
```go
import (
	// ...
	"go.abhg.dev/goldmark/anchor"
	// ...
)

func initGoldmark() goldmark.Markdown {
	return goldmark.New(
		goldmark.WithExtensions(
			// ...
			&anchor.Extender{Texter: anchor.Text("#"), Position: anchor.Before},
		),
		// ...
	)
}
```

And there we go! Now we have links to different sections of our page!

## Lazy Loading Images
This post is pretty big already, but there's _one_ I thing that I'd like to add. If a page happens to contain
a lot of images, I don't want the user to load in all of it at once. So I'll have to add lazy loading to the images, while it's
being converted to markdown. Now there's a simple and a hard way to accomplish this.

One way is to simply run `strings.ReplaceAll(htmlContent, "