Website/items/docs/Walkthrough.md

# Walkthrough

## Summary
* [Overview](#overview)
* [Pipeline definition](#pipeline-definition)
* [Pipeline concurrency](#pipeline-concurrency)
* [Environment variables](#environment-variables)
* [Triggers](#triggers)
* [Steps](#steps)
    * [Fetch and update submodules](#Fetch-and-update-submodules)
    * [Replace hosts and user variables](#replace-hosts-and-user-variables)
    * [Build frontend](#build-frontend)
    * [Check ansible syntax](#check-ansible-syntax)
    * [Apply ansible playbook](#apply-ansible-playbook)
        * [Playbook definition](#playbook-definition)
        * [Delete old spt-item-finder](#delete-old-spt-item-finder)
        * [Copy the project](#copy-the-project)
        * [Copy PHP env file](#copy-php-env-file)
        * [Get JavaScript chunks name](#get-javascript-chunks-name)
        * [Get file names from find output](#get-file-from-find-output)
        * [Copy app.blade.php file](#copy-app-blade-php-file)
        * [Download and install composer dependencies](#download-and-install-composer-dependencies)

## Overview
* The project is split between the frontend and the backend
    * the backend is a [submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) located in [api](../api) that points towards [https://dev.sp-tarkov.com/Rev/spt-items-api.git](https://dev.sp-tarkov.com/Rev/spt-items-api.git)
    * the frontend is a [submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) located in [frontend](../frontend) that points towards [https://dev.sp-tarkov.com/shirito/item-finder-website-frontend.git](https://dev.sp-tarkov.com/shirito/item-finder-website-frontend.git)
* There are two Ansible pipelines
    * A docker pipeline [drone-docker.yml](../drone-docker.yml)
    * A kubernetes pipeline [drone-kubernetes.yml](../drone-kubernetes.yml)
* All ansible playbook files are located in [.ansible](../.ansible)
* The documentation is located in [documentation](../documentation)

## Pipeline definition
```yml
kind: pipeline
type: kubernetes
name: default
```
The pipeline is defined either as Docker or Kubernetes depending on [.drone-docker.yml](../../.drone-docker.yml) or [.drone-kubernetes.yml](../../.drone-kubernetes.yml). The name is set as `default`.

## Pipeline Concurrency
```yml
concurrency:
  limit: 1
```
The pipeline is set to only one build at a time (every subsequent build with be pending).

## Environment variables
```yml
environment:
  SPT_ITEMS_PATH: /var/www/html/aki/spt-item-api
```
Here are the environment variables. They are automatically injected in every step.

## Triggers
```yml
trigger:
  event:
    - push
    - promote
```
The pipeline is run on every push and every promote. Since the repository is *kind of*  a [trunk](https://trunkbaseddevelopment.com), I dont think we need branches policies. Most steps are executed on any push since we want to check that everything builds and is still valid (tests are not added yet). Only the deployment is protected behing the promotion to production.

## Steps
### Fetch and update submodules
```yml
- name: fetch and update submodules to the latest commit
  image: alpine/git
  commands:
  - git submodule init
  - git submodule update --recursive --remote
```
Executed on every push. \
Fetching and updating [submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to the latest commit.

### Replace hosts and user variables
```yml
- name: replace hosts and user variables
  image: ubuntu:impish
  environment:
    DEPLOY_HOST:
      from_secret: deploy_host
    SPT_HOSTNAME:
      from_secret: spt_hostname
    DEPLOYMENT_USER:
      from_secret: deploy_username
  commands:
    -  sed -i 's/{{ SPT_HOSTNAME }}/'"$SPT_HOSTNAME"'/g' ./frontend/.env
    -  sed -i 's/{{ DEPLOY_HOST }}/'"$DEPLOY_HOST"'/g' ./.ansible/inventory
    -  sed -i 's/{{ DEPLOYMENT_USER }}/'"$DEPLOYMENT_USER"'/g' ./.ansible/inventory
```
Executed on every push. \
The following environment variables are injected using Drone secrets:
    * `SPT_HOSTNAME` is used by the frontend to call the backend.
    * `DEPLOY_HOST` is used by Ansible to connect to the remote server via SSH.
    * `DEPLOYMENT_USER` is used by Ansible to connect to the remote server via SSH.
    * all environment variables at the pipeline level (see [Environment variables](#environment-variables))
Using `sed` makes temporary changes in the container/pod instead of commiting secrets in the repo in plain text. \
The changes are never pushed and are discarded when the container/pod is terminated.

### Build frontend
```yml
- name: build frontend
  image: node:lts-alpine3.14
  commands:
    - node -v
    - npm -v
    - yarn --version
    - yarn --cwd ./frontend install
    - yarn --cwd ./frontend build --pure-lockfile
    - rm -rf ./api/public/static/*
    - mv ./frontend/build/* ./api/public
    - rm ./api/public/index.html
```
Executed on every push. \
Since the PHP backend serves the ReactJS frontend, the former is built and moved in the latter.
Notes:
    * `yarn --cwd <path> <command>` executes the command in the specified file
    * `rm -rf ./api/public/static/*` deletes the static files to make sure no old JavaScript files remain
    * `rm ./api/public/index.html` ReactJS is bundled with a `index.html`. It is discarded to use [](https://dev.sp-tarkov.com/Rev/spt-items-api/raw/branch/master/resources/views/app.blade.php) instead.

### Check ansible syntax
```yml
- name: check ansible syntax
  image: plugins/ansible:3
  settings:
    playbook: ./.ansible/playbook.yml
    inventory: ./.ansible/inventory
    galaxy: ./.ansible/requirements.yml
    syntax_check: true
```
Executed on every push. \
Check the Ansible syntax in [playbook.yml](../.ansible/playbook.yml), [inventory](../.ansible/inventory) and [requirements.yml](../.ansible/requirements.yml). The check is executed on every push since we want to detect any error before validating the build using the promotion.

### Apply ansible playbook
```yml
- name: apply ansible playbook
  image: plugins/ansible:3
  settings:
    playbook: ./.ansible/playbook.yml
    inventory: ./.ansible/inventory
    galaxy: ./.ansible/requirements.yml
    private_key:
      from_secret: deploy_ssh_key
  environment:
    DEPLOY_HOST:
      from_secret: deploy_host
    SPT_HOSTNAME:
      from_secret: spt_hostname
    DEPLOYMENT_USER:
      from_secret: deploy_username
  when:
    event:
      - promote
    target:
      - production
```
Executed only on promotion to production. \
This step actually deploys to the server. \
This step is [idempotent](https://en.wikipedia.org/wiki/Idempotence). \
The following environment variables are injected using Drone secrets:
    * `SPT_HOSTNAME` is used by the PHP env file.
    * `DEPLOY_HOST` is used to connect to the remote server via SSH.
    * `DEPLOYMENT_USER` is used to connect to the remote server via SSH.
    * all environment variables at the pipeline level (see [Environment variables](#environment-variables))

#### Playbook definition
```yml
hosts: host
become_user: root
become: true
become_method: sudo
```
Uses the host defined in [inventory](../.ansible/inventory). Remember, the step [Replace hosts and user variables](#replace-hosts-and-user-variables) already replaced the variables at this point. The playbook will be executed as `root` user using `sudo`.

#### Delete old spt-item-finder
```yml
- name: Delete spt-item-finder before adding everything again
  file:
    state: absent
    path: "{{ lookup('env', 'SPT_ITEMS_PATH') }}"
```
Since the copy does not override the folder, this step takes care of it. \
`SPT_ITEMS_PATH` is injected thanks to the pipeline level environment variables (see [Environment variables](#environment-variables))

#### Copy the project
```yml
- name: Copy the project
  copy:
    src: ../api/
    dest: "{{ lookup('env', 'SPT_ITEMS_PATH') }}"
```
Copies the whole project (frontend and backend) from the [api](../api) folder into the server.

#### Copy PHP env file
```yml
- name: Copy PHP .env file
  template:
    src: ./templates/.php-env.j2
    dest: "{{ lookup('env', 'SPT_ITEMS_PATH') }}/.env"
```
Uses [Jinja2](https://jinja2docs.readthedocs.io/en/stable/) to resolve the [template for the PHP .env file](../.ansible/templates/.php_env.j2). \
`SPT_ITEMS_PATH` is injected thanks to the pipeline level environment variables (see [Environment variables](#environment-variables)). \
`SPT_HOSTNAME` is injected in the environments properties (see [Apply ansible playbook](#apply-ansible-playbook))

#### Get JavaScript chunks name
```yml
- name: Get Chunk 2 name
  shell:
    cmd: find "{{ lookup('env', 'SPT_ITEMS_PATH') }}" -type f -name "*chunk.js" -printf "%f\n"
  register: find_output
```
Prepare a find of all JavaScript chunk files for the [app.blade.php.j2](../.ansible/templates/app.blade.php.j2) template. \
`SPT_ITEMS_PATH` is injected thanks to the pipeline level environment variables (see [Environment variables](#environment-variables)).

#### Get file names from find output
```yml
- name: Get file names from find output
  set_fact:
    chunk_list: "{{ find_output['stdout'].split('\n') }}"
```
Splits the string containing the list of all JavaScript chunk files for the [app.blade.php.j2](../.ansible/templates/app.blade.php.j2) template.

#### Copy app.blade.php file
```yml
- name: Copy app.blade.php file
  template:
    src: ./templates/app.blade.php.j2
    dest: "{{ lookup('env', 'SPT_ITEMS_PATH') }}/resources/views/app.blade.php"
```
Uses [Jinja2](https://jinja2docs.readthedocs.io/en/stable/) to resolve the [template for the PHP app.blade.php file](../.ansible/templates/app.blade.php.j2). \
`SPT_ITEMS_PATH` is injected thanks to the pipeline level environment variables (see [Environment variables](#environment-variables)).

#### Download and install composer dependencies
```yml
- name: Download and installs all composer libs and dependencies
  community.general.composer:
    command: install
    working_dir: "{{ lookup('env', 'SPT_ITEMS_PATH') }}"
```

#### Reset files permissions
```yml
- name: Reset files permissions
  file:
    path: "{{ lookup('env', 'SPT_ITEMS_PATH') }}"
    owner: www-data
    group: www-data
    mode: 0744
    recurse: yes
```
Permissions 0644:
    * user: read/write/execute
    * group: read
    * other: read
`www-data` is hardcoded here but it should be the standard user for Apache and Nginx. \
`SPT_ITEMS_PATH` is injected thanks to the pipeline level environment variables (see [Environment variables](#environment-variables)).