git.s-ol.nu mmm / d457233
finish self-hosted s-ol 2 months ago
1 changed file(s) with 182 addition(s) and 3 deletion(s). Raw diff Collapse all Expand all
55 - HTTPS server with multiple subdomains and varying backends
66 - [traefik][traefik] reverse-proxy maintains SSL certificates and serves all requests
77 - [docker-compose][docker-compose] manages running sites/microservices
8 - single DNS wildcard route
98 - a private-public git server
109 - access control, management with [gitolite][gitolite]
1110 - [klaus][klaus] web frontend for browsing and cloning public repos
1211 - fine-grained permissions and SSH public-key access
1312 - micro 'CI' setup rebuilds & redeploys docker images when updates are pushed
1413
15 # HTTPS server
16
14 Most of these projects are very well documented so I won't go into a lot of detail on setting them up.
15
16 # HTTPS Server
17 To run multiple subdomains from a single machine, the HTTP requests need to be matched according to the 'Host' header.
18 Most HTTP servers have good facilities for doing this, e.g. apache has vhosts, nginx has server directives etc.
19
20 In the past I had used apache as my main server, which worked well for static content and PHP apps,
21 but not all applications fit into this scheme too well and configuration is a tad tedious.
22
23 With this latest iteration I am using [traefik][traefik] as a "reverse proxy".
24 This means traefik doesn't serve anything (not even static content) by itself,
25 it just delegates requests to one of multiple configured services based on a system of rules.
26
27 It also handles letsencrypt certificate generation and updates out-of-the-box and can be tightly integrated with docker,
28 so that it automatically reacts to new services being added.
29
30 Traefik can be run at system level, but currently I prefer installing the least system-level applications to have my setup
31 as self-contained as possible. Therefore I went with this [traefik in docker-compose][traefik-in-docker] setup from kilian.io:
32
33 version: '3.4'
34
35 services:
36 traefik:
37 image: traefik:1.5-alpine
38 restart: always
39 ports:
40 - 80:80
41 - 443:443
42 networks:
43 - web
44 volumes:
45 - /var/run/docker.sock:/var/run/docker.sock:ro
46 - ./traefik.toml:/traefik.toml
47 - ./acme.json:/acme.json
48 container_name: traefik
49
50
51 networks:
52 web:
53 external: true
54
55 with the small addition of the `:ro` at the end of the docker socket volume,
56 to prevent attacks on traefik from being able to take over the host docker system (too easily).
57 In the guide you can find more details, including the `traefik.toml` that I am using almost verbatim.
58
59 # Hosting Sites
60 With traefik set up, dockerized services can be added and exposed trivially.
61 For example to start `redirectly`, a tiny link redirect service, this addition suffices:
62
63 redirectly:
64 image: local/redirectly:git
65 restart: always
66 networks:
67 - web
68 labels:
69 - "traefik.frontend.rule=Host:s-ol.nu"
70 - "traefik.enable=true"
71
72 By setting different subdomains in the frontend-rule section, many different services can be provided.
73
74 The image `local/redirectly:git` in this case is built automatically when a repo is pushed (see below).
75
76 note: if a container doesn't have an EXPOSE directive, or EXPOSEs multiple ports,
77 you will have to add a `traefik.port` label specifying which port to use.
78
79 # private/public Git Server
80 While I still have a lot of code on Github, where collaboration is easy,
81 I prefer to own the infrastructure that I store my private projects on.
82 I also wanted to have a public web index of some of the projects.
83
84 The git infrastructure itself is mananged by [gitolite][gitolite], which I really enjoy using.
85
86 repo gitolite-admin @all
87 RW+ = s-ol
88 C = s-ol
89
90 repo public/.*
91 R = @all daemon
92 option writer-is-owner = 1
93
94 repo ... ...
95 RW+ = ludopium
96
97 In the first block I grant myself full access to all repos, as well as the right to automatically create repos by
98 attempting to push/pull from them.
99
100 The second block makes all repos prefixed with `public/` readable by anyone in the gitolite system,
101 as well as the `git-daemon`, which allows cloning via `git://....` access (port 9418).
102 The `write-is-owner` option lets me set the git `description` field using `ssh git@git.s-ol.nu desc`.
103
104 I chose the `public/` prefix because it results in all public repos being stored in one directory together
105 (`/var/lib/gitolite/repositories/public`), where klaus can easily pick them up.
106
107 The klaus web frontend is set up using traefik above like so:
108
109 klaus:
110 image: hiciu/klaus-dockerfile
111 restart: always
112 networks:
113 - web
114 volumes:
115 - /var/lib/gitolite/repositories/public:/srv/git:ro
116 command: /opt/klaus/venv/bin/uwsgi --wsgi-file /opt/klaus/venv/local/lib/python2.7/site-packages/klaus/contrib/wsgi_autoreload.py --http 0.0.0.0:8080 --processes 1 --threads 2
117 environment:
118 KLAUS_REPOS_ROOT: /srv/git
119 KLAUS_SITE_NAME: git.s-ol.nu
120 KAUS_CTAGS_POLICY: tags-and-branches
121 KLAUS_USE_SMARTHTTP: y
122 labels:
123 - "traefik.frontend.rule=Host:git.s-ol.nu"
124 - "traefik.enable=true"
125
126 I am using the ['autoreload' feature][klaus-autoreload] and the `hiciu/klaus-dockerfile` docker image.
127 Setting `KLAUS_USE_SMARTHTTP` allows cloning repos via HTTP.
128
129 In the future I would like to modify klaus a bit, for example by showing the README in the root of a project per default
130 and applying a custom theme.
131
132 # Micro-CI
133 The last piece of the puzzle is automatically deploying projects whenever they are pushed.
134 This can be realized using git's `post-receive` hooks and is generally pretty well known.
135
136 I followed this gitolite guide for [storing repo-specific hooks in the gitolite-admin repo][gitolite-hooks].
137 It requires a change in the gitolite rc file (on the server), but after that you can configure deployment processes in the conf like this:
138
139 @dockerize = public/redirectly ...
140 @jekyllify = blog
141
142 repo @dockerize
143 option hook.post-receive = docker-deploy
144
145 # i actually dont have a jekyll blog anymore but its an easy one as well
146 repo @jekyllify
147 option hook.post-receive = jekyll-deploy
148
149 The hooks are stored in the same repo under `local/hooks/repo-specific`.
150 Here is the `docker-deploy` hook I am using:
151
152 #!/bin/bash
153 read oldrev newrev refname
154 set -e
155
156 # Get project name
157 PROJECT=$(basename "$PWD")
158 PROJECT=${PROJECT%.git}
159
160 # Paths
161 CHECKOUT_DIR=/tmp/git/${PROJECT}
162 TARGET_DIR=... # POINT TO docker-compose.yml directory
163
164 if [ ! -d ${TARGET_DIR} ]; then
165 echo -e "\e[1;32mNo target directory to compile into\e[00m"
166 fi
167
168 mkdir -p ${CHECKOUT_DIR}
169 GIT_WORK_TREE=${CHECKOUT_DIR} git checkout -q -f $newrev
170 echo -e "\e[1;32mChecked out ${PROJECT}.\e[00m"
171
172 cd ${CHECKOUT_DIR}
173 docker build -t local/${PROJECT}:git .
174 echo -e "\e[1;32mImage built.\e[00m"
175
176 cd ${TARGET_DIR}
177 docker-compose up -d ${PROJECT}
178 echo -e "\e[1;32mContainer restarted.\e[00m"
179
180 It will build a `local/$REPO:git` image whenever you push, then run `docker-compose up -d $REPO` in `$TARGET_DIR`.
181 Just make sure that your docker-compose service is called the same as the image name (like in the `redirectly` example above).
182
183 This hook definetely needs an addition to filter which pushes trigger a redeploy (e.g. only tags or only master),
184 but that should be trivial to add (and well documented online).
185
186 ---
187
188 That's basically it!
189 If you have questions or comments i'll be happy to hear from you on twitter, github or [mastodon][merveilles].
17190
18191 [traefik]: https://traefik.io/
19192 [docker-compose]: https://docs.docker.com/compose/
20193 [gitolite]: http://gitolite.com/gitolite/index.html
21194 [klaus]: https://github.com/jonashaag/klaus
195
196 [traefik-in-docker]: https://blog.kilian.io/server-setup/
197 [klaus-autoreload]: https://github.com/jonashaag/klaus/wiki/Autoreloader
198 [gitolite-hooks]: http://gitolite.com/gitolite/cookbook#v36-variation-repo-specific-hooks
199
200 [merveilles]: https://merveilles.town/@s_ol