4090, about 10 seconds to load the weights and another 15 seconds to generate all the completions from the example script

Yikes, thanks

I have a 2060 and I am too afraid and poor to buy a 4090 after import duties and taxes in a tropical country

I could drop the batch size to 5, then the VRAM use seemed to be around 15GB. Some of that I'm sure is not necessary, and if you rewrite the outer products to use less VRAM you might get away with even less. Eventually someone will make a library so you can run it without extra work.

Yeah true, do you think that it a realistic expectation though? I ask this given the events that have led to the leaking of the models. I am genuinely not sure what the optics / real world ramifications are of being publicly associated with projects that leverage models obtained via torrents through either hacking or negligence.

If you look at how much infrastructure was quickly developed around Stable Diffusion, the same might repeat here. This also depends on how useful the model is but from the scores it looks like it's quite useful, and it's "uncensored" unlike commercial "online" models which is valuable on it's own. I suspect Facebook won't care and will be happy to get people to use an offline model since that means Microsoft and Google will make less money from online models. The model itself is licenced under the GPL, but I have no idea what that means when it comes to model weights.

Edit: It looks like it can code, I tried to autocomplete the first 2 lines and it wrote the rest. Local Github Copilot here we come?:

    //find index of element in sorted array in O(log(N)) time using binary search
    int find_idx(int a[N], int element) {
        int low = 0, high = N-1;
        while (low <= high) {
            int mid = (low + high) / 2;
            if (a[mid] == element)
                return mid;
            else if (a[mid] < element)
                low = mid + 1;
            else
                high = mid - 1;
        }
        return -1;
    }