Factories¶
Factories allow you to link multiple bots together and deploy them all from a single command.
You can decide whether to deploy bots locally, or in the AWS account of the invoking user. qalx handles building the entire factory for you and deploying all the bots.
Note
A note on terminology:
Stage: A specific grouping of types (aws, local), sectors and bots that get deployed at the same time. A stage could have multiple sectors of different types, each with multiple bots.
Stack A stack is a collection of sectors of a given type. A single stack could contain multiple sectors with multiple bots. Sectors within a stack are grouped based on the given region_name.
Sector: The specific server that a bot will get deployed to. This could be local for bots that run local to where you run the factory commands, or aws for bots that run in the AWS account of the invoking user. A sector could define some or all of the bots defined in the top level bots key
Workflow: When a bot has finished processing a job you can configure workflows to define which bot(s) the job gets passed to next
A factory consists of multiple stages:
Plan - Codifying your factory in a factory-plan.yml file
Validate - Validating that your factory-plan.yml as the correct format
Pack - Downloading your bot code and preparing for build
Build - Building your factory
Demolish - Demolishing your factory
Stages can be interacted with either via the command line (recommended) or can be invoked via code.
Plan¶
The root of a factory is the factory-plan.yml file. Below is an example configuration. See validate for specific member level validation and description of members
# Namespaced as factory to leave room for extra keys
# at the `bots` level in the future
factory:
# factory names don't have to be unique
name: my_factory_name
# bots are a mapping from the name of the
# bot to details about source/config
bots:
my_bot_1:
source:
url: ssh://fredsmith@bitbucket.org/my-bots-repo
branch: develop
# This will effectively call `bot-start -p 1 bots_package.the_bots:bot1`
bot_path: bots_package.the_bots:bot1
my_bot_2:
source:
path: path/to/local/code
bot_path: bots:bot2
my_bot_3:
source:
pypi: many-useful-bots
bot_path: many_useful_bots:bot_3
my_bot_4:
source:
url: https://github.com/some/repo.git
# This will effectively call `bot-start -p 1 bots_package.the_bots:bot1`
bot_path: bots_package.the_bots:bot4
my_bot_5:
source:
pypi: my-bot-package
version: 1.5.9 # An optional version can be included
bot_path: many_useful_bots:bot_5
# stages are mappings from a stage name to a sector.
# Each sector then specifies the bots running and the arguments
# to pass to `bot.start()
stages:
test:
local_sector:
type: local
bots:
my_bot_1:
# There could be other arguments here that would get passed
# to bot.start()
queue-name: my_bot_1-test-local-queue-name
processes: 1
my_bot_2:
queue-name: my_bot_2-test-local-queue-name
processes: 1
another_local_sector:
type: local
bots:
my_bot_3:
queue-name: my_bot_3-test-local-queue-name
processes: 1
dev:
# Specify a workflow for how your bots interact on this stage
workflow: flow_1
local_sector:
type: local
bots:
my_bot_1:
queue-name: my_bot_1-dev-local-queue-name
processes: 1
another_local_sector:
type: local
bots:
my_bot_3:
queue-name: my_bot_3-dev-local-queue-name
processes: 5
ec2_box_1:
type: aws
# The optional alarm to use for this instance.
alarm: alarm1
parameters:
# The parameters to use when setting up EC2 instances
ImageId: ami-0242408af868
InstanceType: t2.nano
KeyName: keyname # name of key-pair to secure the instance
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-19d539c3545
GroupSet:
# This should be created manually in your AWS account
- sg-8c422def392d
bots:
my_bot_2:
queue-name: my_bot_2-dev-aws-queue-name
processes: 4
ec2_box_2:
type: aws
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.micro
KeyName: keyname
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-19d539c3545
GroupSet:
- sg-8c422def392d
bots:
my_bot_3:
queue-name: my_bot_3-dev-aws-queue-name
processes: 1
ec2_box_3:
type: aws
region_name: eu-west-2
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.nano
KeyName: keyname
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-19d539c3545
GroupSet:
- sg-8c422def392d
bots:
my_bot_3:
queue-name: my_bot_3-dev-aws-queue-name
processes: 1
ec2_box_4:
type: aws
region_name: eu-west-1
parameters:
ImageId: ami-62cee09a7747
InstanceType: t2.nano
KeyName: keyname
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-45f96310c85a
GroupSet:
- sg-0c9c041e6559
bots:
# multiple bots can be started on a single instance
my_bot_3:
queue-name: my_bot_3-dev-aws-queue-name
processes: 1
my_bot_4:
queue-name: my_bot_4-dev-aws-queue-name
processes: 2
prod:
workflow: flow_1
local_connection:
type: local
bots:
my_bot_1:
queue-name: my_bot_1-prod-local-queue-name
processes: 1
ec2_box:
type: aws
region_name: eu-west-2
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.micro
KeyName: keyname
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-19d539c3545
GroupSet:
# This will have to be created manually by the user in their AWS account
- sg-8c422def392d
bots:
my_bot_2:
queue-name: my_bot_2-prod-aws-queue-name
processes: 10
my_bot_3:
queue-name: my_bot_3-prod-aws-queue-name
processes: 10
meta:
key: value
# These get saved on the FactoryBuild object in the API and work like normal tags
tags:
key: value
# The bots that an entity should be passed to once it has finished processing.
# This is restricted to only bots defined on this factory.
workflows:
flow_1:
my_bot_1:
- my_bot_2
- my_bot_3:
my_bot_4
flow_2:
my_bot_1:
my_bot_2:
my_bot_3
# A dictionary of alarms that are available to this factory. Can only
# be applied to AWS sectors. Configuration options match the CloudFormation
# options for a Cloudwatch Alarm. The Dimensions key is not required
# as this is built automatically by pyqalx
alarms:
alarm1:
MetricName: CPUUtilization
Statistic: Average
Period: 60
ComparisonOperator: LessThanThreshold
EvaluationPeriods: 10
Threshold: '5'
Namespace: AWS/EC2
AlarmActions:
- terminate
Aliases¶
Factories use pyyaml under the hood. So you are able to build complex plans using all the power of yaml. A common use case is the ability to reuse certain sections of your plan in order to keep code dry.
Take for example the below:
...snip...
stages:
prod:
ec2_box:
type: aws
region_name: eu-west-2
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.micro
ec2_box2:
type: aws
region_name: eu-west-2
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.micro
...snip...
You could instead take advantage of yaml aliases to avoid having to duplicate the configuration for ec2_box_2. You can even override certain keys
...snip...
stages:
prod:
ec2_box: &my_alias_name
type: aws
region_name: eu-west-2
parameters:
ImageId: ami-3f348b23ee98
InstanceType: t2.micro
ec2_box2:
<<: *my_alias_name
# Start this sector in eu-west-1 but keep
# all other settings the same
region_name: eu-west-1
...snip...
Validate¶
qalx factory-validate --plan=<path-to-your-factory-plan.yml>
Validation is the first stage in building a factory. This ensures that the factory-plan.yml file is well formatted and that the required keys exist.
Why would I want to run this command?
You may want to run the validate command to check that the factories-plan.yml file is structured correctly before
attempting to build your factory
Keys and validation requirements¶
Below is an overview of a sample factories-plan.yml.
factory (required): Root level key
bots (required): A mapping of bot name to source. At least one bot is required. The source could be a private repository, a path to local code or a package name on pypi
- Required sub keys:
source
bot_path
bots:
my_bot_1: # The name of the bot that will be used on this factory.
source:
# A private repository. Provide key:value arguments
# here that should get passed to `git clone`.
# Uses local ssh credentials to authenticate in the same
# way that `git clone git@bitbucket.org/my-bots-repo` does
url: git@bitbucket.org/my-bots-repo
branch: develop
bare: true
# The path to the bot. This should be in the same format
# as when doing `qalx bot-start TARGET
bot_path: bots_package.the_bots:bot1
my_bot_2:
source:
# A bot stored locally
path: path/to/local/code
bot_path: bots:bot2
my_bot_3:
source:
# A package name on pypi
pypi: many-useful-bots
bot_path: many_useful_bots:bot_3
my_bot_4:
source:
url: https://github.com/some/repo.git
bot_path: bots_package.the_bots:bot4
stages (required): A mapping of stages to sector details. You can have as many stages as you want. Stages are useful if you want to test bots on smaller hardware and with less resources or with test data before deploying them to production instances. At least one stage is required. A stage could consist of local and aws sectors.
- Required sub keys for a sector:
type (valid choices local, aws)
bots
# A local stage is one that will deploy bots to the same
# machine as the factory build command is run
stages: # A mapping of stage name to a sector
test: # The name of this stage
workflow: flow_1 # (optional) The name of the workflow this stage should use
local_connection: # The name of the sector
type: local # The type of the sector.
bots: # A mapping of bots that should be active on this sector
my_bot_1:
# A mapping of arguments that get passed to `bot-start`
queue-name: my_bot_1_test_queue
processes: 1
entity-class: dotted.path.to:MyQalxEntityClass
my_bot_2:
queue-name: my_bot_2_test_queue
processes: 1
my_bot_3:
queue-name: my_bot_3_test_queue
processes: 1
# A stage that defines a single AWS sector. This will deploy the bots
# to the users AWS account who invoked the factory build command
stages: # A mapping of stage name to a sector
dev: # The name of this stage
ec2_box_1: # The name of the sector
type: aws # The type of the sector.
alarm: alarm1 # The optional name of the Cloudwatch Alarm to use for this instance
region_name: eu-west-2 # An optional region to start this sector in
parameters:
# The parameters to use when setting up EC2 instances
ImageId: ami-0242408af868
InstanceType: t2.nano
KeyName: keyname # name of key-pair to secure the instance
NetworkInterfaces:
- DeviceIndex: 0
AssociatePublicIpAddress: true
SubnetId: subnet-19d539c3545
GroupSet:
# This should be created manually in your AWS account
- sg-8c422def392d
bots:
my_bot_2:
queue-name: my_bot_2-dev-aws-queue-name
processes: 4
Warning
Ensure that any EC2 Security Groups are created manually before trying to build the factory
meta (not required): A mapping of key:value for the meta keys that should be saved on the factory
tags (not required): A mapping of key:value for the tags that should be saved on the factory. See permissions for more info on tags
workflows (not required): Workflows enable you to define bot(s) that a job should get passed to when a bot has finished processing. Workflows are optional for each stage. Bots defined in workflows must exist on this plan.
- Required sub keys:
bot name
workflows:
flow_1: # The name of the workflow
-my_bot_1: # The bot that should pass the job
# A list of bots that `my_bot_1` will pass the job to when it has
# finished processing. This enables you to fan a job out to multiple bots
- my_bot_2
- my_bot_3
flow_2:
-my_bot_1:
# A mapping of bots that `my_bot_1` will pass the job to when it has
# finished processing.
# This enables you to have a job passed from my_bot_1 -> my_bot_2 -> my_bot_3
-my_bot_2:
- my_bot_3
Pack¶
qalx factory-pack --plan=<path-to-your-factory-plan.yml> (--stage=<stage>) (--no-delete)
Packing is the process by which all the code for the defined bots are downloaded from their various sources to the local computer.
The bots are packed into the value of the FACTORY_PACK_DIR configuration option.
Note
The pack command will automatically call the validate command for you.
--stage is optional - if provided it will only download the bots for the given stage.
If --stage is not provided then all the bots will be downloaded.
--no-delete is optional - By default the code gets deleted after download. Provide this switch to prevent the code from being deleted
Why would I want to run this command?
You may want to run the pack command to check that the credentials are correct for the sources and that
the bots get correctly downloaded.
Sources¶
A bot could come from various sources
Git Repository¶
A remote git repository. Uses your local ssh credentials if necessary to access private repositories.
Warning
A git executable must be available on your system path in order to download from git
# Will use the local ssh credentials to download from
# a private repository
source:
url: git@bitbucket.org/my-bots-repo
# Will download from a public repository
source:
url: https://github.com/some/repo.git
You can provide extra key: value pairs to pass to git in the same way you
would when running git clone from the command line
source:
# Functionally equivalent to
# git clone https://github.com/some/repo.git --branch=develop --quiet --bare
url: https://github.com/some/repo.git
branch: develop
bare: true
quiet: true
Local Code¶
A path to code that is local to the computer the pack command runs on.
source:
path: C:\\Users\\path\\to\\local\\code
Note
Remember to use double slash on Windows.
source:
path: /home/user/path/to/local/code
PyPi¶
The name of a package on pypi
source:
pypi: a-package-name
You can optionally specify a version of a pypi package
source:
pypi: a-package-name
version: 1.2.3
Build¶
qalx factory-build --plan=<path-to-your-factory-plan.yml> --stage=<stage> (--aws-profile=<aws_profile>)
Building is the process by which the packed code for the bots, defined on a specific
stage, is uploaded to qalx, and the bots started on their specific sectors.
Note
Depending on the complexity of your stage and how many remote sectors you have, the build command may take some time to complete. You will be given detailed feedback via the console and log files.
Warning
All bots for a given sector will be started in the same virtual environment. Keep this in mind if certain bots have different dependencies as this could lead to dependency issues.
Note
pyqalx will attempt to install your bot into the virtual environment for all sector types.
For bots with a pypi source it will just install the appropriate file that was downloaded from pypi
during packing - doing the equivalent of pip install <package_name> –upgrade.
For local paths/git sources pyqalx will attempt to install in the following order:
pyproject.toml
setup.py
requirements.txt
For the above it will attempt to do the equivalent of pip install . –upgrade if a pyproject.toml or setup.py is found, and a pip install -r requirements.txt if a requirements.txt file is found
If you don’t want to upgrade a particular package ensure that your dependencies are pinned to the correct version.
Sector Types¶
A sector could be one of the following types:
local - The defined bots will be started on the local machine to where you ran
qalx factory-buildaws - The defined bots will be started on an AWS EC2 instance in your AWS account.
Note
You can have multiple bots defined on an individual sector.
Warning
Sectors with a type of aws will automatically
create a cloudformation stack in your AWS account for each
region_name defined on your factories-plan.yml. The profile used is the one specified in the aws-profile
argument. See aws
for more info on what gets created and the IAM permissions required by the user
who calls qalx factory-build.
local¶
A local sector is one where the defined bots are started on the machine local to where you run
qalx factory-build.
Why would I use a local sector
You may want to use a local sector if you only have a single bot or if you want to test a factory without having to incur additional charges from your cloud hosting provider.
local_sector: # the name of this sector for identification purposes
type: local
bots: # A list of the bots that should be started
my_bot_1: # The bot name for identification purposes
processes: 1
my_bot_2:
# You can pass any argument here that you can pass to `bot-start`
processes: 1
my_bot_3:
processes: 1
aws¶
An AWS sector is one where the defined bots are created on brand new EC2 instances on your
AWS account. The aws-profile optional CLI argument specifies the AWS profile to be used for remote sectors.
The default profile is used if this argument is not provided
Why would I use an aws sector
You may want to use an AWS sector if you have multiple long running bots that require a lot of computing power.
Warning
In order for the factory to be able to build correctly you must specify an ImageId
that is either one of the pyqalx AMIs or uses a pyqalx AMI as a base image. See here for
a list of available pyqalx images.
Warning
You must specify a security group with an outbound rule of HTTPS to 0.0.0.0/0 otherwise the sector won’t be able to contact the API and the build will fail. You will also have to ensure that the sector is placed into a subnet that has access to the internet.
ec2_box_1: # the name of this sector for identification purposes
type: aws
parameters:
# Provider CloudFormation parameters to configure your instance however you wish
ImageId: ami-0242408af868 # The ImageId to use - must be based on a pyqalx image
InstanceType: t2.nano # The InstanceType to use
SecurityGroupIds:
# Security group ID with HTTPS open as an outbound rule
- sg-8c422def392d
bots: # A list of the bots that should be started
my_bot_1: # The bot name for identification purposes
# You can pass any argument here that you can pass to `bot-start`
processes: 4
my_bot_2: # The bot name for identification purposes
# You can pass any argument here that you can pass to `bot-start`
processes: 4
Note
You can add any value to parameters that are defined in the Properties section on the EC2 cloudformation
documentation
Important
You must specify an ImageId and an InstanceType otherwise the AWS sector will
not be able to create the EC2 instances. Valid instance type can be found here
Created Resources
The following resources will be created in your AWS account for an aws sector.
A Cloudformation stack for each region_name defined on the stage
An EC2 instance for each named sector (ec2_box_1, ec2_box_2 etc) on each Cloudformation stack
A secret in Secrets Manager for each EC2 instance and each bot on the instance - this will contain the Qalx
TOKENfrom the config specified on each bot on the sectorA SSM Parameter for all other config values for each bot on the sector
An IAM role for each EC2 instance with permission to access the specific secrets and parameters for the bots on the EC2 instance
Required IAM Permissions
In order to create an aws sector you will need to have an IAM user with at least following permissions.
These minimum permissions will be validated before pyqalx attempts to create anything in your AWS
account.
You may require additional permissions if you provide additional parameters in factories-plan.yaml
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:SimulatePrincipalPolicy",
"Resource": "<your user ARN>"
},
{
"Effect": "Allow",
"Action": [
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:CreateRole",
"iam:DeleteInstanceProfile",
"iam:DeleteRole",
"iam:DeleteRolePolicy",
"iam:GetInstanceProfile",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:PassRole",
"iam:PutRolePolicy",
"iam:RemoveRoleFromInstanceProfile",
"iam:TagRole",
],
"Resource": [
"arn:aws:iam::*:instance-profile/*",
"arn:aws:iam::*:role/*"
]
},
{
"Effect": "Allow",
"Action": [
"cloudformation:CreateStack",
"cloudformation:DeleteStack",
"cloudformation:DescribeStacks",
"cloudformation:DescribeStackEvents",
"cloudformation:UpdateTerminationProtection",
"cloudwatch:PutMetricData"
"ec2:CreateTags",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:TerminateInstances",
"secretsmanager:CreateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:TagResource",
"ssm:AddTagsToResource",
"ssm:DeleteParameter",
"ssm:PutParameter",
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:RunInstances",
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*::image/*",
"arn:aws:ec2:*:*:instance/*",
]
},
{
"Effect": "Allow":,
"Action": [
"cloudwatch:EnableAlarmActions",
"cloudwatch:DeleteAlarms",
"cloudwatch:DisableAlarmActions",
"cloudwatch:PutMetricAlarm"
],
"Resource": [
"arn:aws:cloudwatch:*:*:alarm:*"
]
}
]
}
Extending images
While the provided base images will be useful in most circumstances, there may be times when you need to install
additional software on them to enable your bots to run. You can easily do this by taking the base image, installing your custom software, creating a
new image and then using the new image id in your plan.
If you want to bake python dependencies into the image you will need to install them into the correct virtual environment that pyqalx
uses to bootstrap the image when building the factory.
Note
The bootstrap script will automatically install your bot based on the installing rules when bootstrapping the sectors.
- Linux
user: ec2-user
virtual environment path: /home/ec2-user/pyqalx
- Windows
user: Administrator
virtual environment path: C:UsersAdministrator.venvspyqalxScriptsactivate.bat
Demolish¶
qalx factory-demolish --name=<factory_name>
Demolishing is the process by which your factory gets completely deleted.
Why would I want to run this command?
You may want to run this command if you are finished using your factory and want to remove the resources it is using in order to reduce costs or if you want to make updates to your factory.
The demolish command will initially list all factories matching the factory name as there could be multiple factories with the same name on different stages. Once you have selected the factory that you wish to delete the following will happen:
The factory status will be updated to DELETING
- The bots will be warm terminated according to the specific workflow defined on the stage.
Local bot processes will shut down asynchronously as per normal termination.
Any remote stacks will be deleted (these will take time to delete and will complete asynchronously)
The factory entity will be deleted from the database - bot entities will remain for data integrity purposes
- The local build paths will be deleted if possible
Deleting the local build paths may not be possible if the demolish command was issue from a different physical machine to what the factory was built on
Debug¶
qalx provides various optional debug settings for developing factories. Modifying these settings will help you debug more complex issues with your build
demolish_on_failure¶
Sometimes a factory build will fail. This could be for a number of reasons, perhaps the bot is misconfigured, or there is a problem with a remote stack.
qalx makes every effort to inform you of what went wrong but sometimes you may need to investigate in more detail.
default value
By default, if an error occurs when building a factory, qalx will delete all resources that were created during the build process
example usage
Setting the demolish_on_failure flag to false will stop qalx from deleting the factory entity from the database or deleting any stacks.
Any bots that successfully started will continue to run.
factory:
debug:
demolish_on_failure: false
...the rest of the factory plan here...
Warning
It is up to you to demolish any resources once you are done debugging. The easiest way to do this is with the demolish command, but depending on the reason for the build failure you may need to delete resources manually.
