Patching Projects#

This section solves all problems that arise if you want to customize the actual Pod of a project beyond the limits of what license quotas can do. Also, this customization is specific to individual projects, not across all of them!

To make this possible, there are Licenses which contain a JSON Patch. The basic idea is that this license contains a JSON string of a series of such JSON patch operations, which are aggregated across all licenses (if there is more than one) and applied to the project pod right before it is sent to the Kubernetes API.

This is the sequence leading to the final project pod:

  1. During HELM deployment, the manage/templates/_project.tpl template is rendered and the resulting JSON is stored in the project-pod ConfigMap. If you are audacious, you could deploy your own project-pod ConfigMap with a different template!

  2. When a project is issued to start, the “manage” service reads the template from the project-pod ConfigMap.

  3. It replaces placeholders:

    • {PROJECT_IMAGE} the docker image to use for the project

    • {PROJECT_TAG} the docker image tag to use for the project

    • {project_id} the project’s UUID, as defined by CoCalc in the database

    • {extra_env} additional environment variables to set

    • {project_config} the project configuration, as defined by CoCalc in the database

  4. When the licenses are processed, a quota is computed.

    • This adjust the CPU and memory requests and limits of the project pod.

    • Exact details depend on the max_upgrades and default_quotas server settings.

    • A license quota could contain a “patch” as well.

  5. The combined list of all patches found in the licenses is applied to the project pod template after all the above happened. CoCalc uses the applyPatch function of fast-json-patch.

Example#

As admin, create a “Site License” and in the JSON Patch field, add this:

[
  {
    "op": "add",
    "path": "/metadata/labels/mylabel",
    "value": "test123"
  }
]

This operation adds a label to the metadata of the project pod. See RFC 6902 for more information.

Then, add this license to a project, which will restart it.

As a result, the project pod is now:

$ kubectl get pod -o yaml project-[UUID]
apiVersion: v1
kind: Pod
metadata:
  annotations: {...}
  labels:
    mylabel: test123   # <<< that's new
    network: outside
    project_id: [UUID]
    project_tag: project-20230110-1143
    run: project
[...]

How to create a patch?#

You could the fast-json-patch library to create a patch:

npm init -y
npm i fast-json-patch

Get the current project pod template:

$ kubectl get pod -o json project-[UUID] > p1.json
$ cp p1.json p2.json

Now, edit p2.json to your liking, but don’t touch those fields which are set by Kubernetes! To match the above, e.g. changes like this:

$ diff p1.json p2.json
14c14,15
<             "run": "project"
---
>             "run": "project",
>             "mylabel": "test123"
191,192c192,193
<         "serviceAccountName": "default",
---
>         "serviceAccountName": "default2",

Now, create a script diff.js with the following content:

var jsonpatch = require("fast-json-patch");
var fs = require("fs");

var p1 = JSON.parse(fs.readFileSync("p1.json", "utf8"));
var p2 = JSON.parse(fs.readFileSync("p2.json", "utf8"));

var delta = jsonpatch.compare(p1, p2);
console.log(delta);

and run it:

$ node diff.js
[
  {
    op: 'replace',
    path: '/spec/serviceAccountName',
    value: 'default2'
  },
  { op: 'add', path: '/metadata/labels/mylabel', value: 'test123' }
]

That’s the patch for the license to make the above changes.

Note: under the hood, the patch is stored as a serialized JSON string in the site_licenses database table. e.g.

cocalc=> select * from site_licenses where id = 'f3d76abf-38a5-4e4e-884d-2e70f1afddb9';
id          | f3d76abf-38a5-4e4e-884d-2e70f1afddb9
title       | patch1
activates   | 2023-01-20 16:37:12.595
quota       | {"cpu": 1, "ram": 1, "patch": "[{\"op\":\"add\",\"path\":\"/metadata/labels/mylabel\",\"value\":\"test123\"}]"}
run_limit   | 1
[...]

Problems?#

To debug this, you have to check the log of manage-action-... during project startup. It might fail with an error about a problematic JSON object (e.g. above, it fails for me, because I do not have a “default2” service account).

You can also set the manage.dbg_project_patching value to "1" to output the project pod JSON right before the patch is applied and also the patch set itself. Look for lines containing project_pod= and project_pod_before= and patch=.