Re: [Sks-devel] SKS scaling configuration

[Todd Fleisher]

I don’t know if Kristian chose that number based on actual SKS load, since it 
can be hard to predict how much traffic the various servers in the pool may 
receive at any given time. That being said, the rule of 3 is pretty standard in 
operations to prevent a single point of failure from being revealed in the 
event one of your nodes is unavailable.

-T

> On Mar 4, 2019, at 3:18 PM, Jonathon Weiss <address@hidden> wrote:
> 
> Thanks for the information everyone.  A further question:  I saw the advice 
> of a minimum of three servers.  Anyone know how that was arrived at, or if 
> there is a recommendation on how many queries an individual SKS back-end can 
> handle?
> 
>       Jonathon
> 
>       Jonathon Weiss <address@hidden>
>       MIT/IS&T/Cloud Platforms
> 
> 
> 
> 
> On Fri, 1 Mar 2019, Jeremy T. Bouse wrote:
> 
>> I ended up with the following NGINX configuration...
>> in /etc/nginx/conf.d/upstream.conf:
>> upstream sks_secondary {
>>     server 127.0.0.1:11371 weight=5;
>>     server 172.16.20.52:11371 weight=10;
>>     server 172.16.20.53:11371 weight=10;
>>     server 172.16.20.54:11371 weight=10;
>> }
>> upstream sks_primary {
>>     server 127.0.0.1:11371;
>>     server 172.16.20.52:11371 backup;
>>     server 172.16.20.53:11371 backup;
>>     server 172.16.20.54:11371 backup;
>> }
>> map $arg_op $sks_server {
>>     "stats" sks_primary;
>>     default sks_secondary;
>> }
>> in /etc/nginx/site-available/sks-default:
>> server {
>>     listen 172.16.20.51:80 default_server;
>>     listen 172.16.20.51:11371 default_server;
>>     listen [::]:80 ipv6only=on default_server;
>>     # listen [::]:11371 ipv6only=on default_server;
>>     access_log off;
>>     server_tokens off;
>>     root   /var/www/html;
>>     index  index.html index.htm;
>>     proxy_buffering on;
>>     proxy_buffer_size 1k;
>>     proxy_buffers 24 4k;
>>     proxy_busy_buffers_size 8k;
>>     proxy_max_temp_file_size 2048m;
>>     proxy_temp_file_write_size 32k;
>>     location /pks/hashquery {
>>         proxy_ignore_client_abort on;
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>     }
>>     location /pks/add {
>>         proxy_ignore_client_abort on;
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>         client_max_body_size 8m;
>>     }
>>     location /pks {
>>         proxy_cache ugns_sks_cache;
>>         # proxy_cache_background_update on;
>>         proxy_cache_lock on;
>>         proxy_cache_min_uses 3;
>>         proxy_cache_revalidate on;
>>         proxy_cache_use_stale error timeout updating http_500 http_502 
>> http_503 http_504;
>>         proxy_cache_valid any 10m;
>>         proxy_ignore_client_abort on;
>>         proxy_ignore_headers Cache-Control "Expires";
>>         proxy_pass http://$sks_server;
>>         proxy_set_header Host $host;
>>         proxy_set_header X-Real-IP $remote_addr;
>>         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>>         proxy_set_header X-Forwarded-Proto $scheme;
>>         proxy_set_header X-Forwarded-Port $server_port;
>>         proxy_pass_header Server;
>>         add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>>         add_header X-Proxy-Cache $upstream_cache_status;
>>     }
>> }
>> The NGINX configuration appears to be working fine for me... My 3 backend 
>> nodes are operating as I expect as well.. The problem I'm seeing exhibited 
>> currently is that my primary node which is running along with NGINX seems to 
>> be
>> writing an awful lot of log files and not processing them despite having the 
>> same DB_CONFIG file as the other 3 nodes. With each of the log files running 
>> 100MB it is quickly filling up the drive on that node and then the sks and
>> sks-recon processes simply fall over and crash but the other 3 nodes running 
>> behind it keep chugging along just not getting any further gossip input.
>> I keep seeing the following log entry popping up only on my primary node:
>>     add_keys_merge failed: Eventloop.SigAlarm
>> On 2/25/2019 12:37 PM, Todd Fleisher wrote:
>>            On Feb 23, 2019, at 8:35 PM, Jeremy T. Bouse <address@hidden> 
>> wrote:
>> I didn't have as many locations configured as you show in your example but 
>> it looked like you were defining the map but I didn't see it being used in 
>> any of your location blocks unless I'm missing something. Shouldn't you
>> be using $upstream_server in your proxy_pass configuration?
>> I think you’re on to something here. I just tried commenting out the other 
>> servers from the upstream sks_servers_primary block and am still seeing 
>> stats queries hitting the commented out servers.
>> Kristian - could you please double check the configuration snippets you 
>> provided to me last year and see if something was missing related to this?
>> -T

signature.asc
Description: Message signed with OpenPGP