Re: [Sks-devel] SKS scaling configuration

[Jeremy T. Bouse]

I'd previously had only 2 instances and if they weren't peering outside
and one went down it seemed to cause problems. I went with 3 backend
secondary nodes with the primary node doing the peering outside my
network this time since I was re-deploying from scratch. This way I can
take 1 node out and the cluster still has 3 active nodes. I do have my
primary shutting down to perform key dumps and I also have more of the
traffic for queries go to the secondary nodes and leave the primary node
to handle the peering recon. In fact I just dumped the keys from my
primary node and rebuilt all 4 nodes from the dump last night.

On 3/4/2019 6:18 PM, Jonathon Weiss wrote:
> Thanks for the information everyone.  A further question:  I saw the
> advice of a minimum of three servers.  Anyone know how that was
> arrived at, or if there is a recommendation on how many queries an
> individual SKS back-end can handle?
>
>     Jonathon
>
>     Jonathon Weiss <address@hidden>
>     MIT/IS&T/Cloud Platforms
>
>
>
>
> On Fri, 1 Mar 2019, Jeremy T. Bouse wrote:
>
>>
>> I ended up with the following NGINX configuration...
>>
>> in /etc/nginx/conf.d/upstream.conf:
>>
>> upstream sks_secondary {
>>     server 127.0.0.1:11371 weight=5;
>>     server 172.16.20.52:11371 weight=10;
>>     server 172.16.20.53:11371 weight=10;
>>     server 172.16.20.54:11371 weight=10;
>> }
>>
>> upstream sks_primary {
>>     server 127.0.0.1:11371;
>>     server 172.16.20.52:11371 backup;
>>     server 172.16.20.53:11371 backup;
>>     server 172.16.20.54:11371 backup;
>> }
>>
>> map $arg_op $sks_server {
>>     "stats" sks_primary;
>>     default sks_secondary;
>> }
>>
>> in /etc/nginx/site-available/sks-default:
>>
>> server {
>>     listen 172.16.20.51:80 default_server;
>>     listen 172.16.20.51:11371 default_server;
>>     listen [::]:80 ipv6only=on default_server;
>>     # listen [::]:11371 ipv6only=on default_server;
>>     access_log off;
>>     server_tokens off;
>>     root   /var/www/html;
>>     index  index.html index.htm;
>>
>>     proxy_buffering on;
>>     proxy_buffer_size 1k;
>>     proxy_buffers 24 4k;
>>     proxy_busy_buffers_size 8k;
>>     proxy_max_temp_file_size 2048m;
>>     proxy_temp_file_write_size 32k;
>>
>>     location /pks/hashquery {
>>         proxy_ignore_client_abort on;
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>     }
>>
>>     location /pks/add {
>>         proxy_ignore_client_abort on;
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>         client_max_body_size 8m;
>>     }
>>
>>     location /pks {
>>         proxy_cache ugns_sks_cache;
>>         # proxy_cache_background_update on;
>>         proxy_cache_lock on;
>>         proxy_cache_min_uses 3;
>>         proxy_cache_revalidate on;
>>         proxy_cache_use_stale error timeout updating http_500
>> http_502 http_503 http_504;
>>         proxy_cache_valid any 10m;
>>         proxy_ignore_client_abort on;
>>         proxy_ignore_headers Cache-Control "Expires";
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>         add_header X-Proxy-Cache $upstream_cache_status;
>>     }
>> }
>>
>> The NGINX configuration appears to be working fine for me... My 3
>> backend nodes are operating as I expect as well.. The problem I'm
>> seeing exhibited currently is that my primary node which is running
>> along with NGINX seems to be
>> writing an awful lot of log files and not processing them despite
>> having the same DB_CONFIG file as the other 3 nodes. With each of the
>> log files running 100MB it is quickly filling up the drive on that
>> node and then the sks and
>> sks-recon processes simply fall over and crash but the other 3 nodes
>> running behind it keep chugging along just not getting any further
>> gossip input.
>>
>> I keep seeing the following log entry popping up only on my primary
>> node:
>>
>>     add_keys_merge failed: Eventloop.SigAlarm
>>
>> On 2/25/2019 12:37 PM, Todd Fleisher wrote:
>>             On Feb 23, 2019, at 8:35 PM, Jeremy T. Bouse
>> <address@hidden> wrote:
>>
>> I didn't have as many locations configured as you show in your
>> example but it looked like you were defining the map but I didn't see
>> it being used in any of your location blocks unless I'm missing
>> something. Shouldn't you
>> be using $upstream_server in your proxy_pass configuration?
>>
>> I think you’re on to something here. I just tried commenting out the
>> other servers from the upstream sks_servers_primary block and am
>> still seeing stats queries hitting the commented out servers.
>> Kristian - could you please double check the configuration snippets
>> you provided to me last year and see if something was missing related
>> to this?
>>
>> -T