Decrease Mode and customize rules (#2)

* Service utilization calculation mode to decrease the number of replicas
* Custom max and min percentage for services
* Updated readme
* Fix readme
* Change mean function to median

README.md

@@ -6,6 +6,7 @@
 # Swarm Service Autoscaler
 
 ## Links  
+
 * **[Docker hub](https://hub.docker.com/r/eluki/swarm-service-autoscaler)**
 
 ***
@@ -14,15 +15,15 @@
 
 The project is an application that implements the ability to dynamically change the number of service instances under high load. The application receives all services that have the `swarm.autoscale` label enabled, calculates the average value of the CPU utilization and, based on this, either increases the number of instances or decreases it.
 
-Currently, only the CPU is used for autoscaling in the project. By default, if the CPU load reaches 85%, the service will scale, if it reaches 25%, it will be scaled down.   
+Currently, only the CPU is used for autoscaling in the project. By default, if the CPU load reaches 85%, the service will scale, if it reaches 25%, it will be scaled down.
 But the minimum and maximum values ​​of CPU utilization can be changed through environment variables.
 
 Also, for each service, you can set the maximum and minimum number of replicas to prevent a situation with an uncontrolled increase in the number of replicas (or too much decrease)
 
 ## Usage
 
 1. Deploy Swarm Autoscaler using [`swarm-deploy.yml`](swarm-deploy.yml) from this repository
-2. Add label `swarm.autoscale=true` for services you want to autoscale. 
+2. Add label `swarm.autoscale=true` for services you want to autoscale.
 
 ```yml
 deploy:
@@ -47,8 +48,8 @@ _**The application is configured through environment variables**_
 
 | Setting                     | Default Value      | Description                                                                             |
 | --------------------------- | ------------------ | --------------------------------------------------------------------------------------- |
-| `AUTOSCALER_MIN_PERCENTAGE` | 25.0               | minimum service cpu utilization value in percent (0.0-100.0) for decrease service       |
-| `AUTOSCALER_MAX_PERCENTAGE` | 85.0               | maximum service cpu utilization value in percent (0.0-100.0) for increase service       |
+| `AUTOSCALER_MIN_PERCENTAGE` | 25                 | minimum service cpu utilization value in percent (0-100) for decrease replicas          |
+| `AUTOSCALER_MAX_PERCENTAGE` | 85                 | maximum service cpu utilization value in percent (0-100) for increase replicas          |
 | `AUTOSCALER_DNSNAME`        | `tasks.autoscaler` | swarm service name for in stack communication                                           |
 | `AUTOSCALER_INTERVAL`       | 300                | interval between checks in seconds                                                      |
 | `AUTOSCALER_DRYRUN`         | false              | noop mode for check service functional without enable inc or dec service replicas count |
@@ -57,9 +58,12 @@ _**The application is configured through environment variables**_
 
 _**Services in docker swarm are configured via labels**_
 
-| Setting                                   | Value   | Default | Description                                                                                                                                                                                |
-| ----------------------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `swarm.autoscale`                         | Boolean | `false` | Required. This enables autoscaling for a service. Anything other than `true` will not enable it                                                                                            |
-| `swarm.autoscale.min`                     | Integer | `2`     | Optional. This is the minimum number of replicas wanted for a service. The autoscaler will not downscale below this number                                                                 |
-| `swarm.autoscale.max`                     | Integer | `15`    | Optional. This is the maximum number of replicas wanted for a service. The autoscaler will not scale up past this number                                                                   |
-| `swarm.autoscale.disable-manual-replicas` | Boolean | `false` | Optional. Disable manual control of replicas. It will no longer be possible to manually set the number of replicas more or less than the limit. Anything other than `true` will not enable |
+| Setting                                   | Value   | Default                     | Description                                                                                                                                                                                |
+| ----------------------------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `swarm.autoscale`                         | Boolean | `false`                     | Required. This enables autoscaling for a service. Anything other than `true` will not enable it                                                                                            |
+| `swarm.autoscale.min`                     | Integer | `2`                         | Optional. This is the minimum number of replicas wanted for a service. The autoscaler will not downscale below this number                                                                 |
+| `swarm.autoscale.max`                     | Integer | `15`                        | Optional. This is the maximum number of replicas wanted for a service. The autoscaler will not scale up past this number                                                                   |
+| `swarm.autoscale.disable-manual-replicas` | Boolean | `false`                     | Optional. Disable manual control of replicas. It will no longer be possible to manually set the number of replicas more or less than the limit. Anything other than `true` will not enable |
+| `swarm.autoscale.percentage-max`          | Integer | `AUTOSCALER_MAX_PERCENTAGE` | Optional. Custom maximum service cpu utilization for increase replicas                                                                                                                     |
+| `swarm.autoscale.percentage-min`          | Integer | `AUTOSCALER_MIN_PERCENTAGE` | Optional. Custom minimum service cpu utilization for decrease replicas                                                                                                                     |
+| `swarm.autoscale.decrease-mode`           | String  | `MEDIAN`                    | Optional. Service utilization calculation mode to decrease replicas. Modes: `MEDIAN`, `MAX`                                                                                                |

src/autoscaler_service.py

@@ -4,6 +4,7 @@
 import statistics
 import logging
 from discovery import Discovery
+from decrease_mode_enum import DecreaseModeEnum
 
 from docker_service import DockerService
 
@@ -54,14 +55,20 @@ def __autoscale(self, service):
 
     def __scale(self, service, stats):
         """
-            Method where calculate mean cpu percentage of service replicas and inc or dec replicas count
+            Method where calculate median and max cpu percentage of service replicas and inc or dec replicas count
         """
-        meanCpu = statistics.mean(stats)
+        meanCpu = statistics.median(stats)
+        maxCpu = max(stats)
+
+        serviceMaxPercentage = self.swarmService.getServiceMaxPercentage(service, self.maxPercentage)
+        serviceMinPercentage = self.swarmService.getServiceMinPercentage(service, self.minPercentage)
+        serviceDecreaseMode = self.swarmService.getServiceDecreaseMode(service)
+
         self.logger.debug("Mean cpu for service=%s : %s",service.name,meanCpu)
         try:
-            if(meanCpu > self.maxPercentage):
+            if(meanCpu > serviceMaxPercentage):
                 self.swarmService.scaleService(service, True)
-            elif(meanCpu < self.minPercentage):
+            elif( (meanCpu if serviceDecreaseMode == DecreaseModeEnum.MEDIAN else maxCpu) < serviceMinPercentage):
                 self.swarmService.scaleService(service, False)
             else:
                 self.logger.debug("Service %s not needed to scale", service.name)

src/decrease_mode_enum.py

@@ -0,0 +1,5 @@
+from enum import Enum, auto
+
+class DecreaseModeEnum(Enum):
+    MEDIAN = auto()
+    MAX = auto()

src/docker_service.py

@@ -2,12 +2,16 @@
 import docker
 import logging
 from cache import Cache
+from decrease_mode_enum import DecreaseModeEnum
 
 class DockerService(object):
     AutoscaleLabel = 'swarm.autoscale'
     MaxReplicasLabel = 'swarm.autoscale.max'
     MinReplicasLabel = 'swarm.autoscale.min'
     DisableManualReplicasControlLabel = 'swarm.autoscale.disable-manual-replicas'
+    MaxPercentageLabel = 'swarm.autoscale.percentage-max'
+    MinPercentageLabel = 'swarm.autoscale.percentage-min'
+    DecreaseModeLabel = 'swarm.autoscale.decrease-mode'
 
     def __init__(self, memoryCache: Cache, dryRun: bool):
         self.memoryCache = memoryCache
@@ -50,6 +54,24 @@ def getServiceCpuLimitPercent(self, service):
         except:
             return -1.0
 
+    def getServiceMaxPercentage(self, service, default = None):
+        try:
+            return int(service.attrs.get('Spec').get('Labels').get(self.MaxPercentageLabel))
+        except:
+            return default
+
+    def getServiceMinPercentage(self, service, default = None):
+        try:
+            return int(service.attrs.get('Spec').get('Labels').get(self.MinPercentageLabel))
+        except:
+            return default
+
+    def getServiceDecreaseMode(self, service, default = DecreaseModeEnum.MEDIAN):
+        try:
+            return DecreaseModeEnum[service.attrs.get('Spec').get('Labels').get(self.DecreaseModeLabel).upper()]
+        except:
+            return default
+
     def getContainerCpuStat(self, containerId, cpuLimit):
         containers = self.dockerClient.containers.list(filters={'id':containerId})
         if(len(containers) == 0):